Distinguish state store validation failures from other state store exceptions

Open
  • Is something wrong?
#8,616 1 comment 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
54/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
aws, java
Domain
backend, databases

Research direction

Start by reading the listed state store exception classes and the handling in CompactionTask.commitCompaction; the issue also names CompactionTask.prepareCompactionMessage and CompactionJobDispatcher.dispatch as places to review. Add a shared exception type for the specified validation failures and update commit handling to classify its cause as an invalid job. Done means those failures are handled as described, while batch wrapper exceptions remain independently classified.

Written by the indexing model from the issue text.

Description

needs-refinement under-discussion
User Story

As a developer of Sleeper I want to be able to easily distinguish between state store exceptions that are for validation failures and those that aren't.

Description / Background

When a state store update is invalid against the current state, the state store throws a specific subclass of StateStoreException, e.g. FileNotFoundException, FileReferenceNotFoundException, FileReferenceNotAssignedToJobException, FileAlreadyExistsException. All of these extend StateStoreException directly, as do failures that are not caused by invalid state, such as running out of retries when adding a transaction.

This means code that needs to tell "this update can never succeed" apart from "this update failed and may succeed if retried" has to list the specific exception types it knows about. These lists have drifted apart:

  • CompactionTask.prepareCompactionMessage treats FileReferenceNotFoundException and FileReferenceAssignedToJobException as an invalid job.
  • CompactionTask.commitCompaction treats FileNotFoundException and FileReferenceNotAssignedToJobException (as the cause of the thrown exception) as an invalid job, but not FileReferenceNotFoundException or FileAlreadyExistsException.
  • CompactionJobDispatcher.dispatch treats FileReferenceAssignedToJobException and FileReferenceNotFoundException as unexpected state.

As a result, if a compaction job that has already been committed is committed again synchronously, the second commit fails with FileReferenceNotFoundException. The compaction task does not recognise this as an invalid job, so it returns the message to the queue and counts a consecutive failure. The job is then rerun on each delivery until the message reaches the dead letter queue, and the consecutive failures can terminate a healthy task. See #8418.

Technical Notes / Implementation Details
  1. Add the new exception type and update the exceptions above to extend it.
  2. In CompactionTask.commitCompaction, treat a cause of the new type as an invalid job, replacing the current list of specific exceptions. The message should be deleted from the queue, the failure recorded in the job tracker, and it should not be counted as a consecutive failure.
  3. Consider whether CompactionTask.prepareCompactionMessage and CompactionJobDispatcher.dispatch should also use the new type, or whether they intentionally only handle a subset.

The following should extend it, as retrying the same update will always fail in the same way:

  • FileNotFoundException: input file record is gone, e.g. garbage collected after an earlier commit of the same job.
  • FileReferenceNotFoundException: no reference on the partition, e.g. an earlier commit of the same job removed it (the case in #8418).
  • FileReferenceNotAssignedToJobException: the input reference belongs to a different job or no job, so this commit can never be valid.
  • FileReferenceAssignedToJobException: the reference is already assigned to a job, so it can't be assigned again or split.
  • FileReferenceAlreadyExistsException: a split would add a reference that already exists on the target partition.
  • FileAlreadyExistsException: the file being added, or a compaction's output file, is already in the state store.
  • FileHasReferencesException: the garbage collector can't delete a file that is still referenced.
  • NewReferenceSameAsOldReferenceException: a compaction's output file has the same name as an input, which is invalid whatever the state.
  • InitialiseWithFilesPresentException: partitions can't be initialised when the table has files; not compaction related, but included for consistency.

Exceptions that wrap results of multiple requests, such as ReplaceRequestsFailedException and SplitRequestsFailedException, should not extend it themselves, since a batch may fail for other reasons. Callers can check their cause or their list of failures.

Dominant language
Java
Stars
107
Forks
29
Avg merge
1d 33m
Merged PRs (30d)
115

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gchq/sleeper

All issues in gchq/sleeper

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.