Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Add jitter to the commit retry backoff in `Transaction::commit`

Open Beginner friendly
  • Is something wrong?
#3,348 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
databases

Research direction

Start with Transaction::build_backoff and read the existing Transaction::commit retry tests to understand how retry settings are covered. Enable jitter on the exponential backoff, then verify retries still respect the existing commit.retry.* properties and that the backoff is configured with jitter.

Written by the indexing model from the issue text.

Description

Is your feature request related to a problem or challenge?

Transaction::commit retries retryable errors (such as CatalogCommitConflicts from a REST 409) with exponential backoff built in Transaction::build_backoff:

ExponentialBuilder::new()
    .with_min_delay(Duration::from_millis(props.commit_min_retry_wait_ms()?))
    .with_max_delay(Duration::from_millis(props.commit_max_retry_wait_ms()?))
    .with_total_delay(Some(Duration::from_millis(props.commit_total_retry_timeout_ms()?)))
    .with_max_times(props.commit_num_retries()?)
    .with_factor(2.0)
    .build()

The backoff has no jitter, so every writer waits the same 100 ms, 200 ms, 400 ms, … after a conflict. When several processes append to the same table and conflict on the same commit, they retry in lockstep. Each round lets one writer through and sends the rest into the next round together, so contended writers use up commit.retry.num-retries faster than they would with retries spread out, and the catalog gets bursts of load_table + update_table requests.

We hit this while building an append-only streaming sink where many instances write to one table: in a test with 8 concurrent appenders on the memory catalog, every run had commits that lost to another writer and were retried.

The Java implementation adds jitter to commit retries (Tasks.exponentialBackoff adds a random 0–10% of the delay), so writers from different engines also behave differently on the same table today.

Describe the solution you'd like

Enable jitter in build_backoff, e.g. .with_jitter() on the backon::ExponentialBuilder. It's a one-line change, and the existing commit.retry.* properties keep their meaning.

If matching Java's bounded jitter (up to 10% on top of the delay) is preferred over backon's default jitter, that would also work. The main point is that concurrent writers shouldn't retry at exactly the same moments.

Willingness to contribute

I can contribute a PR for this, including a test.

Dominant language
Rust
Stars
1.4k
Forks
586
Avg merge
2d 12h
Merged PRs (30d)
67

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg-rust

All issues in apache/iceberg-rust

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.