Flink: JdbcLockFactory concurrent lock-table creation can fail with "table already exists" and restart the job
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
Research direction
Read JdbcLockFactory.initializeLockTables() and the CREATE_LOCK_TABLE_SQL definition, then check how TriggerManager and LockRemover call lockFactory.open(). Reproduce the concurrent table creation path and inspect the related Flink maintenance tests. Done means both operators can open the factory when the lock table is initially absent without failing the job.
Written by the indexing model from the issue text.
Description
Apache Iceberg version
1.12.0 (latest release)
Query engine
Flink
Please describe the bug 🐞
With flink-maintenance.lock.jdbc.init-lock-table=true, two operators in the same job open the same JdbcLockFactory and each tries to create the lock table:
- TriggerManager calls lockFactory.open() in initializeState.
- LockRemover calls lockFactory.open() in open() (LockRemover.java:114).
JdbcLockFactory.initializeLockTables() runs CREATE_LOCK_TABLE_SQL only if the table is missing. That statement is a plain CREATE TABLE without IF NOT EXISTS, so the sequence is check-then-create. If both operators run the
check before either one has created the table, the second CREATE TABLE fails and the task goes to FAILED.
Encountered this bug when writing an E2E test in PR,
https://github.com/apache/iceberg/pull/18661#discussion_r4470462231
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
- Dominant language
- Java
- Stars
- 9.4k
- Forks
- 3.6k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 138
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iceberg
-
Core: TableMetadata.Builder allows adding encryption keys to v1/v2 tablesPossibly taken @yusufyildiz34 claimed this 10 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg#18632 · 1 comment ·
Maintainers usually reply within 1 day
-
Kafka: Filter empty strings and whitespace in IcebergSinkConfig stringToListPossibly taken @mohamed-elbrik claimed this 15 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Parquet TIME_MILLIS statistics are not converted to microsecondsPossibly taken @chenjian2664 claimed this 23 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Temporal transforms put pre-epoch timestamps at `.999999` into the previous unitPossibly taken @gitedmond claimed this 31 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
apache/iceberg#18371 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
-
OpenAPI: Remove deprecated TrueExpression and FalseExpressionPossibly taken @zhang-arvin claimed this 26 days ago. Openimprovement
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/iceberg#18351 · 3 comments ·
Maintainers usually reply within 1 day
Similar issues
-
needs-area-label
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 2 days
-
enhancement Java macOS
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
freerouting/freerouting#1079 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/maven-enforcer#1074 ·
Maintainers usually reply within 1 day
-
Gradle liquibaseDiffChangelog drops jhi_date_time_wrapperPossibly taken @mraible claimed this today. Openarea: triage theme: undefined
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
jhipster/generator-jhipster#35998 ·
Maintainers usually reply within 1 day