Garbage collector ignores the per-invocation file limit when deletions fail

Open Beginner friendly
  • Is something wrong?
#8,631 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
85/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
aws, java
Domain
backend, cloud

Research direction

Start with GarbageCollector.deleteInBatches, TableFilesDeleted, and the property description for sleeper.gc.files.maximum. In GarbageCollectorTest, add the described case alongside shouldDeleteFilesUpToLimitWithinSingleInvocation: use five ready-for-GC files and a delete fake that fails every file, with the limit set to two. Done means the invocation attempts at most the limit plus any files in the current batch, and the property description explains that failed files count.

Written by the indexing model from the issue text.

Description

bug needs-refinement under-discussion
Description / Background

The property sleeper.gc.files.maximum limits how many files the garbage collector deletes per table in one invocation. Its description says the limit exists to avoid reaching the lambda timeout.

GarbageCollector.deleteInBatches keeps reading ready-for-GC files while deleted.getDeletedFileCount() < maxFiles. Only files that were successfully deleted from S3 increase that count. Files that fail to delete are recorded with TableFilesDeleted.failed and don't count.

If S3 deletes fail for every file, e.g. because of a permissions or bucket policy problem, the count stays at zero. One invocation then sends a delete request for every ready-for-GC file in the table, however many there are. A large table can run until the lambda times out. Failed files stay ready for GC, so every later invocation does the same.

Steps to reproduce
  1. In GarbageCollectorTest, set sleeper.gc.files.maximum to 2 and sleeper.gc.batch.size to 1.
  2. Create 5 files that are ready for GC, and use a DeleteFiles fake that fails every file.
  3. Run the garbage collector. All 5 files are attempted, not 2.
Expected behaviour

Every file the garbage collector attempts counts towards sleeper.gc.files.maximum, whether it was deleted or not. An invocation stops after attempting that many files, apart from finishing the current batch, as the property description says.

Technical Notes / Implementation Details
  • Add a count of attempted files to TableFilesDeleted, or count files as deleteInBatches reads them, and use that in the loop condition. Keep getDeletedFileCount for the return value and the logs.
  • Update the property description to say failed files count towards the limit.
  • Add a test to GarbageCollectorTest for the steps above, alongside shouldDeleteFilesUpToLimitWithinSingleInvocation.
Dominant language
Java
Stars
107
Forks
29
Avg merge
1d 33m
Merged PRs (30d)
115

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gchq/sleeper

All issues in gchq/sleeper

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.