Transaction log follower can trigger after a transaction was deleted

Open
  • Is something wrong?
#8,485 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
aws, java
Domain
backend, distributed-systems

Research direction

Start by reading the related issues #8348 and #4218, which the issue identifies as connected to resolving this problem. The issue describes a lagging follower processing a transaction after it has been deleted, leaving the job tracker behind the state store. Done means resolving how to handle this case so the job tracker is eventually updated without falling further behind; the issue does not name implementation files or tests.

Written by the indexing model from the issue text.

Description

bug under-discussion
Description / Background

In a table with a large number of partitions, sometimes compactions can be successfully committed to the state store, but remain uncommitted indefinitely according to the job tracker and reporting.

The follower lambda, which records the commit in the job tracker, can get so far behind that the transaction is deleted before it is processed in the follower.

Steps to reproduce
  1. Create a Sleeper table with a very large number of partitions
  2. Ingest enough data to create compactions across all partitions
  3. Leave this running with new data coming in
  4. See some compactions are stuck uncommitted in compaction job reports, even though they are committed to the state store
Expected behaviour

The job tracker should eventually be updated. It shouldn't get further and further behind.

Technical Notes / Implementation Details

There are some issues related to this:

We need to resolve how to deal with this in those.

Dominant language
Java
Stars
107
Forks
29
Avg merge
1d 33m
Merged PRs (30d)
115

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gchq/sleeper

All issues in gchq/sleeper

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.