Wrong results and panics after a column is dropped and added back under the same name

Aperta
  • Qualcosa non va?
#57 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
32/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust, sql
Ambito
data, database

Direzione di ricerca

Start in crates/datafusion/src/table_provider/catalog_table_provider.rs, where the Arrow schema is cached when the provider is built, and in crates/datafusion/src/read/scan.rs, where columns are selected by name. Reproduce by creating a table, inserting a row, dropping and re-adding a column with a different type, then querying through a provider built before the change. Done means the stale-provider query either matches a fresh session or fails with a clear schema-changed error, with no wrong counts or panics. The issue leaves the fix open, so agree on the approach with maintainers before coding.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug
Describe the bug

IcebergCatalogTableProvider caches the table's Arrow schema when it is built (catalog_table_provider.rs#L81-L84), and IcebergCatalogProvider keeps each provider for its lifetime (schema.rs#L64-L94). Scans select columns by name from the snapshot they read (read/scan.rs#L93-L97).

Dropping a column and adding one back under the same name is how a column's type is changed when the change isn't an allowed promotion. After that, a provider built before the change reads the new column but still declares the old column's type and nullability. DataFusion trusts the declared schema:

  • It folds price IS NULL to false because the cached price is NOT NULL. The plan replaces the scan with EmptyRelation, so the count is 0 without reading the table.
  • Operators that rely on the declared Float64 panic on the decimal arrays.
  • SELECT * fails because a column declared non-nullable contains nulls.
To Reproduce
  1. Create ns.t(id INT NOT NULL, price DOUBLE NOT NULL), register an IcebergCatalogProvider as c in session S, and run INSERT INTO c.ns.t VALUES (1, 1.5).
  2. Outside S, change price to a decimal: drop it with update_schema().delete_column("price"), then add it back with update_schema().add_column(AddColumn::optional("price", Type::Primitive(PrimitiveType::Decimal { precision: 10, scale: 2 }))). From Spark, that is ALTER TABLE t DROP COLUMN price followed by ALTER TABLE t ADD COLUMN price DECIMAL(10, 2).
  3. Insert (2, 2.25) from a session that registers the catalog after step 2.
  4. Run these queries in S and, for comparison, in a new session:
Query Session S New session
SELECT count(*) FROM c.ns.t WHERE price IS NULL 0 1
SELECT count(price) FROM c.ns.t panics: primitive array 1
SELECT id, price * 2 FROM c.ns.t panics: primitive array (1, NULL), (2, 4.50)
SELECT * FROM c.ns.t error: Column 'price' is declared as non-nullable but contains null values (1, NULL), (2, 2.25)

The panics come from arrow-array's as_primitive (cast.rs:849) and reach the caller as a Join Error.

If price is added back as DOUBLE instead, S still counts 0 and still fails SELECT *. Only the panics need the type change.

Expected behavior

S returns the same results as the new session, or fails every query on t with an error saying that the table's schema changed after the provider was built. It never returns a wrong count or panics.

Additional context
  • This is another concrete case of #24 §2, on the read path. #56 is the write-path counterpart.
  • #42 is related but different: there the provider declares the table's current schema while the scan binds to an older snapshot's schema.
  • #48 lets Ballista rebuild a provider with the schema the client planned against, so a rebuilt provider behaves the same way.
  • Possible fixes:
    • in scan, check each declared field against the field of the same name in the scanned snapshot's schema (field id from PARQUET:field_id, type and nullability), and return an error on a mismatch
    • resolve the table, and with it the schema, per query (#24)
Lingua principale
Rust
Stelle
30
Fork
30
Merge medio
3g 19h
PR unite (30g)
15

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-iceberg

Tutte le issue di apache/datafusion-iceberg

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.