[Feature Request]: Allow non-identifier column names in Python Row
メンテナーはふだん 1 日以内に返信
@schizophrenicmaniac がすでに取り組んでいます。
2026年10月20日 から。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 48/100
- issue の種類
- 機能追加
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
調査の方向性
Start in sdks/python/apache_beam/typehints/schemas.py at SchemaTranslation.named_tuple_from_schema, then read the RowCoder construction path in sdks/python/apache_beam/coders/coder_impl.py and the tests covering schema translation and row coding. Reproduce the failure with the supplied schema containing EXPR$0 and 31-60 days; done means Python can construct and encode or decode rows while preserving those field names.
索引モデルが issue の本文から書いたものです。
説明
What would you like to happen?
What happened?
In the Beam Portability API (schema.proto) and the Java SDK (org.apache.beam.sdk.schemas.Schema / org.apache.beam.sdk.values.Row), schema field names can be arbitrary strings. For example, Calcite SQL (SqlTransform) routinely produces output schemas with column names such as:
- Unaliased expression names:
EXPR$0,EXPR$1 - Quoted SQL column aliases containing spaces, punctuation, or leading digits:
>120 days,31-60 days,91-120 days(e.g., TPC-DS Q62, Q99) - Python reserved keywords:
from,class,return
However, when a schema is decoded in the Python SDK (for instance, when receiving the output of cross-language SqlTransform or any portable RowCoder), SchemaTranslation.named_tuple_from_schema (sdks/python/apache_beam/typehints/schemas.py) attempts to construct a typing.NamedTuple (or dataclasses.make_dataclass when a field starts with _, thanks to a recent patch in #40542):
if any(name.startswith('_') for name, _ in subfields):
user_type = dataclasses.make_dataclass(
type_name,
subfields,
namespace={'_asdict': _dataclass_asdict},
frozen=True)
else:
user_type = NamedTuple(type_name, subfields)
Both typing.NamedTuple and dataclasses.make_dataclass enforce strict Python identifier rules (name.isidentifier() and not keyword.iskeyword(name)), causing RowCoder initialization to fail with:
ValueError: Type names and field names must be valid identifiers: 'EXPR$0'
or:
TypeError: Field names must be valid identifiers: '31-60 days'
Why is this a feature gap?
- Cross-language incompatibility: A valid
PCollection<Row>produced by a Java transform (such asSqlTransformor xlang I/O connectors) cannot be consumed in Python if any column name is not a strict Python identifier. - Incremental workarounds in
named_tuple_from_schema: Previously,dataclasses.make_dataclasswas added as a fallback specifically for leading-underscore column names like_change_type(CDC metadata), becauseNamedTuplerejected them. However,make_dataclassstill rejects non-identifier characters ($, spaces,-,>), leading digits, dunder names, and Python keywords. - Python runtime already supports arbitrary attribute names:
RowCoderImpl(sdks/python/apache_beam/coders/coder_impl.py) encodes and decodes fields usinggetattr(value, name)and positional constructionself.constructor(*values). In Python,getattrand class namespaces (type(name, (tuple,), namespace)withproperty(operator.itemgetter(i))) work with arbitrary string keys, not just valid Python identifiers.
Steps to Reproduce
from apache_beam.coders import RowCoder
from apache_beam.portability.api import schema_pb2
schema = schema_pb2.Schema(
id="sql_expr_schema",
fields=[
schema_pb2.Field(
name="EXPR$0",
type=schema_pb2.FieldType(atomic_type=schema_pb2.INT64)),
schema_pb2.Field(
name="31-60 days",
type=schema_pb2.FieldType(atomic_type=schema_pb2.INT64)),
])
coder = RowCoder(schema) # Raises ValueError: Type names and field names must be valid identifiers: 'EXPR$0'
Proposed Solution
In SchemaTranslation.named_tuple_from_schema (sdks/python/apache_beam/typehints/schemas.py), when any field name in subfields is not a valid Python identifier or is a Python keyword (i.e., rejected by both NamedTuple and make_dataclass), construct a dynamic tuple-backed row class via type(type_name, (tuple,), namespace) that:
- Sets
_fields = tuple(name for name, _ in subfields)and__annotations__ = dict(subfields)somatch_is_named_tuple()andRowTypeConstraint.from_user_type()continue to work seamlessly. - Binds
property(operator.itemgetter(i))in the class dictionary for each field name (which Python allows for arbitrary string keys, sogetattr(row, "EXPR$0")andgetattr(row, "31-60 days")work in O(1) time). - Implements
__new__(cls, *args, **kwargs),_asdict(), and__repr__()matchingNamedTuplesemantics while preserving the exact portable schema field names.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Prism Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
- 主要言語
- Java
- スター
- 8.7k
- フォーク
- 4.8k
- 平均マージ
- 2日 12時間
- マージ済み PR(30日)
- 371
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
apache/beam のほかの issue
-
awaiting triage bug P3
難易度 1/5 1時間未満 初心者へのやさしさ 67/100
apache/beam#41197 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
[Bug]: [Python] WriteToJson silently rounds floats to 10 decimal places対応中かも @Abhi2939 が 2 日前に担当しました。 オープンio P2 python yaml
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
apache/beam#41102 · コメント 4 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
apache/beam#39624 · リアクション 2 件 ·
メンテナーはふだん 1 日以内に返信
-
[Bug]: PubsubIO used in batch incorrect batch cutoff size対応中かも @1fanwang が 79 日前に担当しました。 オープンbug io P3 pinned pubsub
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
apache/beam#28011 · コメント 4 件 ·
メンテナーはふだん 1 日以内に返信
-
build P3 sub-task
難易度 1/5 1時間未満 初心者へのやさしさ 68/100
メンテナーはふだん 1 日以内に返信
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
spAnser/reacharound#38 ·
-
難易度 1/5 1時間未満 初心者へのやさしさ 89/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
-
Type: Bug
難易度 2/5 1〜3時間 初心者へのやさしさ 83/100
MovingBlocks/Terasology#5415 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100