Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Expose Python UDF inlining as a public, reusable extension codec

Ouverte
#1,704 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
48/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
python, rust

Piste de recherche

Commencez par crates/core/src/codec.rs et les points d’entrée PythonLogicalCodec, PythonPhysicalCodec et SessionContext.with_python_udf_inlining. Lisez la dépendance #1703 et les changements de composition des codecs de #1678 avant de déterminer comment le codec public doit être installé dans la chaîne. Le travail est terminé lorsque des consommateurs Rust externes peuvent ajouter le codec public et préserver le format filaire Python UDF existant sans réimplémenter le cadrage de son payload.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

enhancement rust

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

datafusion-python can serialize a Python scalar UDF inline, by cloudpickling the function together with its schemas and volatility, so a plan referencing a Python UDF round-trips into a process that does not have that function registered. This is what SessionContext.with_python_udf_inlining turns on.

That capability is not reachable from outside the crate. PythonFunctionScalarUDF is pub(crate), and encode_python_scalar_udf / decode_python_scalar_udf in crates/core/src/codec.rs are private; only the try_encode_python_scalar_udf / try_decode_python_scalar_udf wrappers are pub(crate), and they are called from the two codecs this crate installs on every session. A Rust project that already has its own physical extension codec, and wants Python UDFs to travel through it, has to reimplement the cloudpickle framing to stay wire-compatible. Requested in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-5100366976.

Describe the solution you'd like

Extract the inlining behavior into its own LogicalExtensionCodec / PhysicalExtensionCodec implementation and make that type public, rather than exporting two free functions and a struct. Since #1678 made codecs compose, with_python_udf_inlining can then install that codec into the chain instead of being a flag consulted inside PythonLogicalCodec and PythonPhysicalCodec, and an outside consumer can put the same type in its own chain and get the same wire format for free. Suggested independently in https://github.com/apache/datafusion-python/pull/1678#discussion_r3918287124.

A public codec is a better thing to commit to than the current internals. It hides the payload layout, which is versioned and should stay free to change, and it exposes the behavior at the granularity DataFusion already understands.

Describe alternatives you've considered

Marking the existing functions and PythonFunctionScalarUDF pub as they stand. Smaller, but it publishes the payload layout as API, and it does not compose with anything — a caller still has to decide where in its own codec to call them.

Additional context

Depends on #1703, gating pyo3/extension-module, since the crate cannot currently be linked as a plain Rust dependency. Follow-up from #1678.

Langage dominant
Python
Étoiles
605
Forks
176
Merge moyen
1 j 23 h
PR mergées (30 j)
8

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/datafusion-python

Toutes les issues de apache/datafusion-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.