Postpone-bucket getWriteId misparses the write id when data-file.prefix or the commit user contains the -s- separator
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- java
- Domain
- data-engineering
Research direction
Start at PostponeBucketFileStoreWrite.getWriteId and reproduce the postpone-bucket path with a data-file.prefix or commit user containing -s-. Trace the filename parsing around the -w- marker, then verify that the extracted write id routes files correctly through getWriteId(...) % parallelism and that malformed names still fail as expected.
Written by the indexing model from the issue text.
Description
Search before asking
- I searched in the issues and found no similar issues.
Paimon version
master (1.5-SNAPSHOT)
Compute Engine
Flink (postpone-bucket write path).
Minimal reproduce step
- Create a postpone-bucket table and set
data-file.prefixto a value that contains the-s-separator (for examplea-s-b). - Write data so postpone-bucket data files are produced. Each file name is
{data-file.prefix}-u-{commitUser}-s-{writeId}-w-{uuid}-{count}. - Read or compact the table, which routes splits by
getWriteId(fileName) % parallelism.
What doesn't meet your expectations?
PostponeBucketFileStoreWrite.getWriteId recovered the write id by splitting on the first -s- in the file name. When data-file.prefix (or a custom commit user) already contains -s-, the first -s- is not the writer's separator, so the parse either throws Data file name ... does not match the pattern or returns a wrong write id. A wrong id then routes the split to the wrong reader under getWriteId(...) % parallelism.
Anything else?
The correct anchor is the -w- writer marker that always terminates the interpolated prefix, so the write id is the segment between the last -s- before -w- and -w- itself.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 472
Getting set up
We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/paimon
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
REST Catalog alterTable silently reports success when the server returns a non-table/column 404Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
beehive-lab/TornadoVM#1151 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
FasterXML/jackson-dataformats-binary#823 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day