Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug]: After 0.9, disabling self-optimizing leaves process RUNNING instead of CLOSED

Aperta
  • Qualcosa non va?
#4,431 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@zhang-arvin ci sta già lavorando.

Dal 9/10/2026.

  • #4432 di @zhang-arvin — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
java, spark
Ambito
backend, databases

Direzione di ricerca

Start by reading TableRuntimeRefreshExecutor.handleConfigChanged, DefaultOptimizingService.cancelProcess, and TableController.cancelOptimizingProcess, then trace how process state and process_failed_count are persisted in table_process. Reproduce disabling self-optimizing and cancelling an orphan RUNNING process, and add or run tests for both flows. Done means the DB row closes even when the in-memory process is absent, failed cancellation is not reported as HTTP 200, and user/config closure is not counted as a process failure.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

type:bug
What happened?

After upgrading AMS from 0.8.x to 0.9.x, disabling self-optimizing no longer closes the current optimizing process.

In 0.8.x, after self-optimizing.enabled=false the process became CLOSED. In 0.9.x the row stays RUNNING in table_process (often for days), the dashboard keeps showing RUNNING, and Dashboard Cancel does not change it.

We also see amoro_table_optimizing_process_failed_count_total increase in the same window. close() / failed commit both call completeProcess(false), so turning optimizing off is counted as a process failure. There is no metric that means “CLOSED by user/config” vs “FAILED commit”.

Expected

  • Setting self-optimizing.enabled=false should persist the current process as CLOSED (same as 0.8 / AMORO-2250), including when the in-memory OptimizingProcess is already gone.
  • Cancel should close by processId from DB. If it cannot close, the HTTP API should not return 200.
  • Close/cancel should not increment process_failed_count (or should use a separate reason/metric).
Affects Versions

0.9.0-incubating (regression vs 0.8.x)

What table formats are you seeing the problem on?

Iceberg

What engines are you seeing the problem on?

AMS, Spark

How to reproduce
  1. Iceberg table with a MINOR process in RUNNING.
  2. From Spark (Copy-on-Write upsert job):
ALTER TABLE catalog.db.table SET TBLPROPERTIES ('self-optimizing.enabled' = 'false');
  1. Optionally run Spark CoW upsert on the same table (common production pattern: disable optimizing, then rewrite).
  2. Check AMS DB / Optimizing page:
SELECT process_id, status, process_type, create_time, finish_time
FROM table_process
WHERE table_id = ? AND status = 'RUNNING';

The process stays RUNNING (finish_time NULL). Repeating this daily accumulates many orphan RUNNING rows (we had 15+ on one table). status='CLOSED' / 'FAILED' can be empty.

  1. Open the process on the Optimizing page and click Cancel. UI still shows RUNNING. HTTP cancel returns 200.
Relevant log output
When Spark already rewrote the same files and AMS later tries to commit the leftover process:


org.apache.amoro.exception.OptimizingCommitException: unexpected commit error
Caused by: org.apache.iceberg.exceptions.ValidationException: Missing required files to delete: ...
	at org.apache.iceberg.ManifestFilterManager.validateRequiredDeletes
	at org.apache.amoro.server.optimizing.UnKeyedTableCommit.rewriteFiles
	at org.apache.amoro.server.optimizing.OptimizingQueue$TableOptimizingProcess.commit


That path increments `process_failed_count`. A successful in-memory `close(false)` also increments it via `persistAndSetCompleted(false)`.
Anything else

This looks like a 0.9 process-framework regression of #2250 (disable self-optimizing should close the running process). 0.9 moved history to table_process and only closes the in-memory process:

  • TableRuntimeRefreshExecutor.handleConfigChanged closes only when config flips true→false and getOptimizingProcess() != null. Later refreshes do not retry. If the in-memory process is already null, the DB row stays RUNNING.
  • DefaultOptimizingService.cancelProcess returns false when getOptimizingProcess() is null or the processId does not match. TableController.cancelOptimizingProcess still returns HTTP 200.

This is different from #4172 (all tasks SUCCESS, stuck before COMMITTING). Here the process is an orphan RUNNING after disable/cancel, and failed_count treats close as failure.

Suggested direction (no code in this issue, just for maintainers):

  1. When self-optimizing.enabled=false, close all RUNNING processes for that table from DB, not only the in-memory one; retry on every refresh while disabled.
  2. cancelProcess(processId) should persist-close the table_process row if the in-memory process is missing; HTTP should fail when cancel did not happen.
  3. completeProcess(false) after user/config close should not increment process_failed_count.
Are you willing to submit a PR?
  • Yes I am willing to submit a PR!
Code of Conduct
  • I agree to follow this project's Code of Conduct
Lingua principale
Java
Stelle
1.2k
Fork
398
Merge medio
3g 11h
PR unite (30g)
23

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/amoro

Tutte le issue di apache/amoro

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.