apache/pulsar

Make the autoSkipNonRecoverableData can work for schema Ledger

Open

#17,221 opened on Aug 23, 2022

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Java (13,697 stars) (3,485 forks)batch import
Stalehelp wanted

Description

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

After we enabled autoSkipNonRecoverableData, the topic will skip the non-recoverable ledger which will lose some data, but the topic is able to load and continues to provide messaging service. This usually happens when multiple bookie replicas are corrupted, or the Ledger has been deleted This usually happens when multiple bookie copies are corrupted

But the Ledger used by the schema is not able to be skipped even if autoSkipNonRecoverableData=true

Solution

Make the schema can skip the non-recoverable Ledger if autoSkipNonRecoverableData=true And looks like we need to provide a new option to skip the data that the schema is lost.

Alternatives

No

Anything else?

No

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Contributor guide

Tech stack
java
Domain
backenddata
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Understanding of Apache Pulsar schema ledgerJava development experience
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
Examine the current implementation of autoSkipNonRecoverableData for topics in the Pulsar broker. Identify how schema ledgers are handled during recovery and where the skip logic is missing. Consider adding a configuration option to enable schema ledger skipping. Look at files like ManagedLedgerImpl.java and SchemaStorage.java. Check the assignee's recent activity on this issue for any updates.