Compaction task retries a job whose input files no longer exist

Open
  • Is something wrong?
#8,418 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
aws, java, rust
Domain
backend, data-engineering

Research direction

Start in CompactionTask.commitCompaction and CompactionTask.prepareCompactionMessage, where FileReferenceNotFoundException is handled differently. Then read FileReferenceStoreQueries.isAssigned and the existing tests CompactionTaskCommitTest and CompactionTaskAssignFilesTest. Done when a re-delivered committed job is deleted without rerunning or counting as a failure, and the output file is not rewritten. The ECSCompactionTaskRunnerLocalStackIT test should also pass.

Written by the indexing model from the issue text.

Description

bug needs-refinement under-discussion
Background / Description

A compaction job message can be delivered to a compaction task again after the job has already been committed to the state store. For example:

  • The task crashes after committing the job, and before deleting the message from the job queue.
  • With asynchronous commit, the task crashes after sending the commit request, and before deleting the message.
  • The keep alive thread in SqsCompactionQueueHandler fails to extend the message visibility while the job is running, so another task picks up the message.

Once a job has been committed, its input files have no references. The garbage collector deletes them from S3 after the GC delay (sleeper.table.gc.delay.minutes, 15 minutes by default).

If the job is delivered again after its input files have been deleted, the compaction fails reading them from S3. CompactionTask.processCompactionMessage handles this as a generic failure, so the message is returned to the queue and a consecutive failure is counted. This happens with both synchronous and asynchronous commit.

The job can never succeed, but it is retried on every delivery until the message reaches the dead letter queue. Each attempt counts as a consecutive failure, which can terminate a healthy task.

Steps to reproduce

In CompactionTaskCommitTest, with either value of COMPACTION_JOB_COMMIT_ASYNC:

  1. Run a job and let it commit.
  2. Delete the input files, as if the garbage collector had run.
  3. Send the same job to the queue again and run the task.

Result: the job fails reading its input files, the message is returned to the queue, and a consecutive failure is counted.

Expected behaviour

If a compaction fails because an input file does not exist, the task deletes the message, records the failure in the job tracker, and does not count it as a consecutive failure.

Technical Notes / Implementation Details
  1. In both the Java and DataFusion compaction runners, detect when a compaction fails because an input file does not exist, and throw a specific exception type for it. This keeps CompactionTask independent of S3 and Hadoop exception types.
  2. In CompactionTask.processCompactionMessage, catch that exception, delete the message from the queue and record the job as failed, without incrementing the consecutive failures.

To be agreed: whether a missing input file should ever be treated as a fault that needs investigation, e.g. a symptom of a garbage collector bug. The proposal here is that the failure recorded in the job tracker is enough, and the message does not need to go to the dead letter queue.

Out of scope

If the job is delivered again before its input files are deleted, the compaction reruns and overwrites its output file. The task's handling of the resulting commit failure is covered by #8616, and safe reads during an overwrite by #8617 .

Dominant language
Java
Stars
107
Forks
29
Avg merge
1d 33m
Merged PRs (30d)
115

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gchq/sleeper

All issues in gchq/sleeper

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.