[Bug]: Beam SQL Parquet tables fail on TIME columns: Unhandled logical type beam:logical_type:time:v1

Offen
  • Stimmt etwas nicht?
#40,994 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
48/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
java, sql
Bereich
backend, databases

Rechercherichtung

Start with AvroUtils.getFieldSchema and genericFromBeamField, where the issue identifies the missing TIME identifier handling, and compare them with the DATE conversion paths. Read the Beam SQL TIME mapping in CalciteUtils and inspect the Avro time-millis/time-micros conversions. Add a JUnit regression test under sdks/java/extensions/sql using the DirectRunner repro; done means TIME values can be written to and read back from a Parquet table. The Avro time unit choice needs maintainer guidance.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

awaiting triage bug java P2
What happened?

A Beam SQL Parquet table (CREATE EXTERNAL TABLE ... TYPE parquet) with a TIME column can't be written. On master (b0c68c4a4f7), the INSERT fails with:

java.lang.RuntimeException: Unhandled logical type beam:logical_type:time:v1

Minimal repro, run as a JUnit test in sdks/java/extensions/sql with the DirectRunner (p is a TestPipeline):

BeamSqlEnv env = BeamSqlEnv.inMemory(new ParquetTableProvider());
env.executeDdl("CREATE EXTERNAL TABLE t (c TIME) TYPE parquet LOCATION '/tmp/time-table/'");
BeamSqlRelUtils.toPCollection(p, env.parseQuery("INSERT INTO t VALUES (TIME '12:34:56')"));
p.run().waitUntilFinish(); // RuntimeException: Unhandled logical type beam:logical_type:time:v1

Calling AvroUtils.toAvroSchema on the table's schema, which is what ParquetTable does (L60, L69, L111), throws the same exception.

Cause. The two sides use different identifiers for TIME:

  • Beam SQL maps TIME to FieldType.logicalType(SqlTypes.TIME), whose identifier is beam:logical_type:time:v1 (CalciteUtils L114).
  • AvroUtils.getFieldSchema only recognizes the legacy "TIME" identifier (L1224). Every other identifier falls through to the Unhandled logical type exception (L1242).
  • genericFromBeamField has the same gap (L1389, L1406).

DATE does handle both identifiers: "DATE".equals(identifier) || SqlTypes.DATE.getIdentifier().equals(identifier) (L1220). That's why DATE columns work and TIME columns don't.

Expected: TIME columns can be written and read back. The CREATE EXTERNAL TABLE docs list DATE, TIME, TIMESTAMP as supported Parquet types.

A fix probably needs a design call. Should SqlTypes.TIME (a java.time.LocalTime) map to Avro time-micros (long) or time-millis (int)? Reading the column back also has to convert to LocalTime, and the time-millis/time-micros branches of the Avro-to-Beam conversion currently return the raw int/long. I'm happy to send a fix once a maintainer confirms which mapping they'd like.

For reference, here is what AvroUtils.toAvroSchema produces today for each type, from a temporary test that runs CREATE EXTERNAL TABLE ... TYPE parquet for each column type:

Beam SQL type Beam field type Avro type
TINYINT, SMALLINT, INTEGER BYTE, INT16, INT32 int
BIGINT INT64 long
FLOAT FLOAT float
DOUBLE DOUBLE double
DECIMAL DECIMAL bytes (decimal)
BOOLEAN BOOLEAN boolean
DATE SqlTypes.DATE int (date)
TIME SqlTypes.TIME exception
TIMESTAMP DATETIME long (timestamp-millis)
CHAR, VARCHAR STRING string
BINARY, VARBINARY BYTES bytes

Several rows of the docs' Parquet type table don't match this: the int types, FLOAT, DATE and TIME. I'll file that docs correction separately.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components
  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
Vorherrschende Sprache
Java
Sterne
8.7k
Forks
4.8k
Ø Merge
2 T. 12 Std.
Gemergte PRs (30 T.)
371

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/beam

Alle Issues in apache/beam

Ähnliche Issues

Weitere Issues zu Java

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.