[Bug] Shredded Variant files miss internal field IDs and cannot be read by Java Paimon

Search before asking

• I searched in the issues and found nothing similar.

Paimon-cpp version

• Apache Paimon C++ v0.3.0 (efbfc848)
• The problem is also present on current main (21228d92b30cc7e987239f0dbca4371120bdfb0b).
• Reader used to reproduce the failure: Apache Paimon Java 1.4.2.

Minimal reproduce step

1. Create a Paimon table containing a VARIANT column.

2. Write at least one data file with paimon-cpp and enable Variant shredding, either with a configured shredding schema or with schema inference. A minimal typed projection such as the following is sufficient:

   payload VARIANT
     typed_value ROW<age INT, city STRING>
   

3. Inspect the Parquet schema written by paimon-cpp. The outer Paimon table field has a field ID, but the generated Variant physical fields (metadata, value, typed_value) and descendants of typed_value do not have Parquet field IDs.

4. Read the file through Apache Paimon Java 1.4.2, for example from Spark.

The Java reader fails while constructing the Variant read plan, before reading any row:

java.lang.NullPointerException: Cannot invoke
"org.apache.parquet.schema.Type$ID.intValue()" because the return value of
"org.apache.parquet.schema.Type.getId()" is null
    at org.apache.paimon.format.parquet.ParquetSchemaConverter.convertToPaimonField(ParquetSchemaConverter.java:386)
    at org.apache.paimon.format.parquet.ParquetSchemaConverter.convertToPaimonField(ParquetSchemaConverter.java:406)
    at org.apache.paimon.format.parquet.VariantUtils.variantFileType(VariantUtils.java:49)

The existing VariantParquetTest.ShreddedWriteAndReadRoundTrip can also expose the problem by opening the raw Parquet footer and asserting that every generated field in the shredded Variant subtree has a field ID. The current C++-write/C++-read round trip succeeds because it does not verify Java interoperability or these footer IDs.

What doesn't meet your expectations?

Expected behavior

A shredded Variant file written by paimon-cpp should be readable by the corresponding Java Paimon implementation. Its generated physical schema should carry deterministic Paimon/Parquet field IDs compatible with the schema produced by the Java writer.

Actual behavior

paimon-cpp can write and read the file itself, but Java Paimon cannot open it. ParquetSchemaConverter recursively converts the typed_value subtree and requires every converted Parquet field to have an ID. It dereferences a null Type.getId() for the first generated field without one.

The source of the mismatch appears to be:

• VariantShreddingSchemaImpl creates metadata, value, typed_value, nested object fields, and list elements with plain arrow::field(...), without paimon.id metadata.
• VariantShreddingWritePlan installs this generated physical type while preserving only the outer table field metadata.
• ParquetFieldIdConverter copies an existing paimon.id to PARQUET:field_id; it does not assign IDs when the generated Arrow field has none.

Configured shredding, inferred per-file shredding, and adaptive shredding all use this physical-schema construction path, so all modes can produce files that Java Paimon cannot read when typed_value is present.

This blocks cross-language interoperability. For example, an engine using paimon-cpp for native writes currently has to fall back to the Java writer for shredded Variant data, otherwise Java/Spark readers cannot consume the resulting files.

Anything else?

The ordinary unshredded Variant layout does not have this problem: UnshreddedStructType explicitly assigns IDs 0 and 1 to value and metadata.

Although Parquet field IDs are optional in the file-format specification and Variant physical members are identified by name, the current Java Paimon reader uses its generic Parquet-to-Paimon schema converter for typed_value, and that converter requires IDs. Therefore the file is not interoperable with the Java implementation that paimon-cpp targets.

Suggested fix and regression coverage:

1. Assign deterministic paimon.id metadata to all generated shredded Variant fields, including nested object fields and list/map descendants, matching Java Paimon's physical schema conventions.
2. Ensure the IDs survive configured, inferred, and adaptive shredding paths and are emitted as Parquet field IDs.
3. Extend the C++ Parquet test to assert the raw footer IDs for a nested shredded schema.
4. Add a cross-language regression test that writes a shredded Variant file with paimon-cpp and reads it with Java Paimon (and ideally the reverse direction).

Are you willing to submit a PR?

• I'm willing to submit a PR!