Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Replace the logical codec's object registry with durable metadata

Ouverte
#1,724 1 commentaire 0 réactions 1 personne assignée Voir sur GitHub

@s5dsn-eqee y travaille déjà.

Depuis le 21/9/2026.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
74/100
Type d'issue
Refactorisation
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python, rust
Domaine
api, backend, testing

Piste de recherche

Commencez dans examples/datafusion-ffi-example/src/logical_extension_codec.rs, au niveau de try_encode_table_provider, et comparez l’approche IPC dans examples/distributed/storage-library/src/codec.rs. Mettez à jour python/tests/_test_logical_extension_codec.py, notamment l’assertion concernant l’ancien codec et un test de décodage deux fois. C’est terminé lorsque le registre et le code de token ont disparu, que provider_prefix est utilisé, que les tests spécifiés passent et que grep -in token ne renvoie rien.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

enhancement rust

examples/datafusion-ffi-example/src/logical_extension_codec.rs parks live table providers in a process-global HashMap and encodes an integer token into it. Encoding inserts, decoding removes, so the same bytes cannot be decoded twice, one plan cannot fan out to several readers, and a plan that never reaches a decoder keeps its provider alive for the life of the process. extension-guide/codecs.md tells authors not to do this.

Unlike the physical codec in the same crate (see the quarantine sub-issue), this one is fixable: try_encode_table_provider at line 148 claims node.downcast_ref::<MemTable>(), which is narrow, and a MemTable is fully describable by its schema and batches.

Pattern to copy: examples/distributed/storage-library/src/codec.rs — same Arrow IPC technique, same error convention (internal_datafusion_err! on encode, since this process holds the object; exec_datafusion_err! on decode, since those are foreign bytes).

Verified prerequisites: MemTable.batches is pub (datafusion-catalog/src/memory/table.rs:69), typed Vec<PartitionData> where PartitionData = Arc<tokio::sync::RwLock<Vec<RecordBatch>>>. MemTable::try_new rejects zero partitions (table.rs:84). arrow is already a dependency with IPC available, so no Cargo.toml change.

Proposed wire format, keeping the per-instance prefix the dispatch tests rely on:

<provider_prefix> | b"MEMTBL1" | u32 LE n_partitions | { u32 LE ipc_len | ipc stream }*

One stream per partition, because MemTable partition boundaries become output partitions. A stream carries its schema even when empty, so an empty partition round-trips.

Two traps worth writing down before someone hits them:

  • Use try_read() on each partition lock, not blocking_read(). The FFI codec runs with a tokio runtime handle installed, and blocking_read panics in that context.
  • On decode, build MemTable::try_new from the IPC schema, not the schema: SchemaRef argument. try_new validates schema.contains(&batch.schema()), so metadata drift would surface as a spurious mismatch. This is the opposite choice from storage-library/src/codec.rs:376-380, which must honour the plan's schema because it re-reads files from disk; here the batches are the payload. Worth a comment noting the contrast, since the two codecs otherwise look alike.

Done when: the registry, token_id(), and the HashMap/Mutex/OnceLock/AtomicU64 imports are gone; the struct field token is renamed provider_prefix to match the Python kwarg that already uses that name; and grep -in token over the file returns nothing.

Tests: of 19 tests in python/tests/_test_logical_extension_codec.py, one changes. test_installing_a_codec_cannot_hijack_an_earlier_codecs_objects asserts len(before) == len(after) with a comment about tokens being minted per encode; that comment becomes false and the assertion becomes weaker than reality, so it should become assert before == after. Add one test for the property the guide claims and nothing currently covers: encode once, decode twice on one session, assert both produce the same rows. All 47 tests in the planner crate should be unaffected — every assertion there is on call counters, never on payload shape.

Langage dominant
Python
Étoiles
605
Forks
176
Merge moyen
1 j 23 h
PR mergées (30 j)
8

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/datafusion-python

Toutes les issues de apache/datafusion-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.