Spark RewriteManifestsSparkAction sortBy throws "AnalysisException: [FIELD_NOT_FOUND] No such struct field" for nested field in partition spec

Open Beginner friendly
#17,425 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
java, spark
Domain
databases

Research direction

Start in spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java at sortColumn, especially around line 320, and reproduce the Spark 4.1 rewrite manifests procedure with a nested partition field whose flattened name contains a dot. Done means sorting by that partition field no longer raises FIELD_NOT_FOUND.

Written by the indexing model from the issue text.

Description

bug
Apache Iceberg version

1.11.0 (latest release)

Query engine

Spark

Please describe the bug 🐞

In Spark 4.1, calling rewrite manifests procedure with sort by on a partition field that have a dot in the name after being flattened (for example a nested field in a struct), will throw this exception
AnalysisException: [FIELD_NOT_FOUND] No such struct field a in a.b
This is because in RewriteManifestsSpark::sortColumn builds the Spark column reference for a custom clustering field by string concatenation — col("data_file.partition." + partitionFieldName), we could fix it by quotting partitionFieldName

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Dominant language
Java
Stars
9.3k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
143

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg

All issues in apache/iceberg

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.