DataFusion reads of a data file may mix two versions of the file if it is overwritten during the read

Open
  • Is something wrong?
#8,617 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
aws, java, postgresql, rust
Domain
backend, databases

Research direction

Start by reading the objectstore_ext wrappers and the code that configures stores for DataFusion queries and compactions; check how those stores compose with ReadaheadStore. Add unit coverage for overwrite detection between HEAD and ranged GET, and for unchanged objects. The issue also proposes a Java integration test extending LocalAwsTestBase against Floci; done means DataFusion fails cleanly on an overwrite or returns rows from one consistent version.

Written by the indexing model from the issue text.

Description

needs-refinement under-discussion
User Story

As a user of Sleeper, I want parts of the system that use DataFusion to be resilient against a file begin updated in S3 mid read.

Background / Description

A data file can be overwritten in S3 while it is being read, when a committed compaction job is run again (see #8418).

DataFusion reads a Parquet file with several requests: a HEAD for the size (retrieve_object_metas), a GET for the footer, then ranged GETs for the data. None of these pin the object version. The parquet crate's ParquetObjectReader sets no if_match, and Sleeper's wrappers in objectstore_ext pass the options through unchanged.

If the file is overwritten mid-read, the reader can mix one version's footer with the other's data. This usually causes an error, but could occasionally return incorrect rows without one. This affects queries and compactions through DataFusion. Reads through Hadoop S3A and the shared Parquet metadata cache are not affected, as both detect the change.

Reads through DataFusion should fail cleanly if a file is overwritten during the read, rather than mixing data from two versions of the file.

Technical Notes / Implementation Details
  1. Add an object store wrapper in objectstore_ext, or extend an existing one, to remember the ETag returned by the first HEAD or GET request for each object, and set GetOptions::if_match to that ETag on all later GET requests for the same object. If the object changed, S3 will reject the request with a precondition failure.
  2. Make sure this is applied to the stores used for both queries and compactions, and works together with ReadaheadStore.

Optional: retry the read of a file when it fails with a precondition failure, since the new version of the file is still valid and referenced in the state store. This could also be left to the caller, e.g. retrying the whole query.

Tests to add
  • Unit, objectstore_ext: with the wrapper applied, overwrite an object between a HEAD and a ranged GET. Expect the GET to fail with a precondition failure.
  • Unit, objectstore_ext: with no overwrite, ranged GETs succeed and the ETag condition does not cause extra requests.
  • IT, DataFusion against Floci: overwrite a Parquet file during a leaf partition query and expect the query to fail cleanly, or succeed with one consistent version if a retry is added. The Rust code has no existing tests against a local AWS container, so this is likely easiest as a Java test extending LocalAwsTestBase that runs the query through DataFusion.

Floci (floci/floci:2.3.0, as used by SleeperLocalAwsContainer) supports this: a GET with a stale If-Match fails with PreconditionFailed for byte ranges, suffix ranges and whole objects, including multipart uploads.

Dominant language
Java
Stars
107
Forks
29
Avg merge
1d 33m
Merged PRs (30d)
115

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gchq/sleeper

All issues in gchq/sleeper

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.