Substrait consumer doesn't recognise `shift_left` / `shift_right` from functions_arithmetic.yaml

Offen Anfängerfreundlich
  • Stimmt etwas nicht?
#27,059 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@jonasHanhan arbeitet bereits daran.

Seit 04.11.2026.

  • #27072 von @jonasHanhan — offen

Bewertung

Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Anfängerfreundlichkeit
76/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
rust
Bereich
backend

Rechercherichtung

Beginne mit datafusion/substrait/src/logical_plan/consumer/expr/scalar_function.rs, wo der Issue besagt, dass der Consumer derzeit nur die bitwise_-Namen zuordnet; prüfe die benachbarten Zuordnungen und Tests. Verifiziere, dass standardmäßige Aufrufe von shift_left und shift_right den Shift-Operatoren von DataFusion zugeordnet werden, und prüfe auch die Namensgebung des Producers. Das Ziel ist erreicht, wenn die Standardnamen im reproduzierten Plan funktionieren und shift_right_unsigned einen expliziten Fehler für nicht unterstützte Funktionen meldet.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

bug
Describe the bug

I found this inconsistency while checking DataFusion against a formalization of Substrait plan semantics, when plans that call Substrait's standard shift functions were rejected by from_substrait_plan.

Substrait's functions_arithmetic.yaml defines shift_left, shift_right and shift_right_unsigned (added in Substrait v0.74.0, substrait-io/substrait#799). DataFusion's consumer maps the names bitwise_shift_left and bitwise_shift_right to its shift operators, and rejects a plan that calls the standard names with Unsupported function name: "shift_left". The producer writes the same non-standard names, so a DataFusion plan with a << 1 calls bitwise_shift_left, which other consumers won't find in the extension. The current names may simply predate the standard functions.

To Reproduce

DataFusion 55.0.0 (Python bindings, datafusion==55.0.0, built on DataFusion 55.1.0); the same on 51.0.0. The consumer's mapping on main still lists only the bitwise_ names (datafusion/substrait/src/logical_plan/consumer/expr/scalar_function.rs).

# uv run --no-project --with datafusion==55.0.0 --with "substrait>=0.37" python DF-10.py
import json
import datafusion
from datafusion import SessionContext
from datafusion.substrait import Consumer, Producer, Serde
from google.protobuf import json_format
from substrait.plan_pb2 import Plan

print("datafusion", datafusion.__version__)
ctx = SessionContext()

# SELECT shift_left(a, 1), shift_right(a, 1) FROM (VALUES (CAST(5 AS INT))) t(a),
# written with the functions of Substrait's functions_arithmetic.yaml.
urn = "extension:io.substrait:functions_arithmetic"
def call(anchor):
    return {"scalarFunction": {"functionReference": anchor,
        "outputType": {"i32": {"nullability": "NULLABILITY_REQUIRED"}},
        "arguments": [
            {"value": {"selection": {"directReference": {"structField": {"field": 0}}, "rootReference": {}}}},
            {"value": {"literal": {"i32": 1}}}]}}
plan = {
  "extensionUrns": [{"extensionUrnAnchor": 1, "urn": urn}],
  "extensions": [
    {"extensionFunction": {"extensionUrnReference": 1, "functionAnchor": 1, "name": "shift_left:i32_i32"}},
    {"extensionFunction": {"extensionUrnReference": 1, "functionAnchor": 2, "name": "shift_right:i32_i32"}}],
  "relations": [{"root": {"names": ["l", "r"], "input": {"project": {
    "common": {"emit": {"outputMapping": [1, 2]}},
    "input": {"read": {
      "baseSchema": {"names": ["a"], "struct": {"types": [{"i32": {"nullability": "NULLABILITY_REQUIRED"}}], "nullability": "NULLABILITY_REQUIRED"}},
      "virtualTable": {"expressions": [{"fields": [{"literal": {"i32": 5}}]}]}}},
    "expressions": [call(1), call(2)]}}}}]}
msg = json_format.ParseDict(plan, Plan())
try:
    lp = Consumer.from_substrait_plan(ctx, Serde.deserialize_bytes(msg.SerializeToString()))
    print(ctx.create_dataframe_from_logical_plan(lp).to_pydict())
except Exception as e:
    print("consumer error:", e)

# The producer, for its part, names these functions bitwise_shift_left / bitwise_shift_right.
ctx.sql("CREATE TABLE t (a INT) AS VALUES (5)").collect()
p = Producer.to_substrait_plan(ctx.sql("SELECT a << 1 AS l, a >> 1 AS r FROM t").logical_plan(), ctx)
out = Plan(); out.ParseFromString(p.encode())
print("producer extensions:", [e.extension_function.name for e in out.extensions])

Output:

datafusion 55.0.0
consumer error: DataFusion error: This feature is not implemented: Unsupported function name: "shift_left"
producer extensions: ['bitwise_shift_left', 'bitwise_shift_right']

With the two names changed to bitwise_shift_left and bitwise_shift_right, the same plan runs and returns {'l': [10], 'r': [2]}, so only the name is in the way.

Expected behavior

The consumer accepts shift_left and shift_right and maps them to Operator::BitwiseShiftLeft and Operator::BitwiseShiftRight, so the plan above returns {'l': [10], 'r': [2]}. The producer could write the standard names too, as was done for is_nan, sign and others in #16949. shift_right_unsigned has no DataFusion operator, so an explicit "not supported" error would fit it.

Additional context
Willingness to contribute

I would like to contribute a fix for this bug, but request guidance from the DataFusion community

Vorherrschende Sprache
Rust
Sterne
9.6k
Forks
2.6k
Ø Merge
3 T. 10 Std.
Gemergte PRs (30 T.)
326

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/datafusion

Alle Issues in apache/datafusion

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.