[Bug] PyPaimon nested projection returns incorrect MAP values and nullability

Start by running the PyPaimon reproduction with the PyArrow and Parquet setup described, using CatalogFactory, Schema, and the projection reader entry points. Trace how MAP selectors become struct fields and how ROW nullability is propagated; done means literal keys such as .foo, selector-prefix cases, and nullable projected fields behave correctly for both Parquet and row files.

Search before asking

• I searched existing issues for these projection failures and found no matching report.

Paimon version

Master at bb5498ff3, using paimon-python from the source checkout.

Compute Engine

PyPaimon, Python 3.12.6, PyArrow 19.0.1, macOS. The example uses a local filesystem table with Parquet files.

Minimal reproduce step

Read two distinct MAP keys, foo and .foo, from a one-row table:

import tempfile

import pyarrow as pa

from pypaimon import CatalogFactory, Schema


with tempfile.TemporaryDirectory() as warehouse:
    catalog = CatalogFactory.create({"warehouse": warehouse})
    catalog.create_database("default", False)
    data = pa.table(
        {
            "attrs": pa.array(
                [[("foo", 100), (".foo", 107)]],
                type=pa.map_(pa.string(), pa.int64()),
            )
        }
    )
    catalog.create_table(
        "default.repro",
        Schema.from_pyarrow_schema(
            data.schema, options={"bucket": "-1", "file.format": "parquet"}
        ),
        False,
    )
    table = catalog.get_table("default.repro")
    wb = table.new_batch_write_builder()
    writer = wb.new_write()
    try:
        writer.write_arrow(data)
        wb.new_commit().commit(writer.prepare_commit())
    finally:
        writer.close()

    rb = table.new_read_builder().with_projection(["attrs['foo']", "attrs['.foo']"])
    result = rb.new_read().to_arrow(rb.new_scan().plan().splits()).to_pydict()
    print(result)
    assert result == {"attrs_foo": [100], "attrs__foo": [107]}

What doesn't meet your expectations?

Expected:

{'attrs_foo': [100], 'attrs__foo': [107]}

Actual:

{'attrs_foo': [100], 'attrs__foo': [100]}

The read returns the value for foo in both columns without reporting an error. Reading only attrs['.foo'] raises ArrowInvalid. I reproduced both cases with Parquet and row files.

Anything else?

I also found two projection failures in the same checkout:

• MAP selector prefix matching: with MAP columns named attrs and attrs['x, and attrs containing ('x[0]', 42), attrs['x[0]'] drops the requested column, while attrs["x[0]"] returns 42. A longer field-name prefix wins before the remaining selector is validated.
• ROW nullability: for a nullable r: ROW<x BIGINT NOT NULL>, writing r = NULL and r = {x: 7} then projecting r.x returns [None, 7] with a not null output field. Writing this Arrow result to Parquet raises Column 'r_x' is declared non-nullable but contains nulls.

For the first example, the selected MAP keys become struct fields during reading. Passing .foo as a string to pyarrow.compute.struct_field interprets it as a field path. The projection reader needs to preserve the literal key name.

Are you willing to submit a PR?

• I'm willing to submit a PR!