apache/pulsar

The retention is not working due to non closing ledger

Open

#6,935 opened on May 10, 2020

View on GitHub
 (10 comments) (0 reactions) (0 assignees)Java (3,485 forks)batch import
area/brokerhelp wantedlifecycle/staletype/bug

Repository metrics

Stars
 (13,697 stars)
PR merge metrics
 (Avg merge 3d 10h) (146 merged PRs in 30d)

Description

Describe the bug

I have been researching the retention problem and high disk usage. Currently the retention is defined to 2 hours, ttl 600 sec. "managedLedgerMaxLedgerRolloverTimeMinutes" : "240", "managedLedgerCursorRolloverTimeInSeconds" : "14400" "managedLedgerCursorMaxEntriesPerLedger" : "50000" "managedLedgerMaxEntriesPerLedger" : "50000",

pulsar version is 2.5.0

if i run bin/pulsar-admin topics stats-internal persistent://internal/eyecloud/test591-InputEvents i get the following output

{ "entriesAddedCounter" : 75562, "numberOfEntries" : 75562, "totalSize" : 8430787149, "currentLedgerEntries" : 75562, "currentLedgerSize" : 8430787149, "lastLedgerCreatedTimestamp" : "2020-05-09T15:19:16.289Z", "waitingCursorsCount" : 3, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "1073:75561", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 1073, "entries" : 0, "size" : 0, "offloaded" : false } ], "cursors" : { "InputEvents-SourceReader" : { "markDeletePosition" : "1073:75561", "readPosition" : "1073:75562", "waitingReadOp" : false, "pendingReadOps" : 0, "messagesConsumedCounter" : 75562, "cursorLedger" : 1075, "cursorLedgerLastEntry" : 428, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2020-05-09T15:19:16.292Z", "state" : "Open", "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "properties" : { } }

If i understand the output correctly , the ledger was opened almost 1 day ago, it has more then 50000 entries and according to the policies should be closed.

This is relatively a big topic and allocated lot of disk space. i suspect that it's not deleted because the ledger is still opened. I see the same things on the other similar topics . Is my assumption correct ?

Contributor guide