Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Feature] Add query-scoped planning limits and cooperative cancellation

Aperta
  • Qualcosa non va?
#435 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
data-engineering

Direzione di ricerca

This is design work, so start by reading the API surfaces in include/paimon/scan_context.h and include/paimon/table/source/table_scan.h, then the limit ordering in src/paimon/core/table/source/data_table_batch_scan.cpp and the manifest fan-out in src/paimon/core/operation/file_store_scan.cpp (ReadFileEntries). The first step is agreeing the API shape with maintainers on this thread. Done means an approved contract plus tests that inject allocation, I/O and submission failures.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Parent integration: sirius-db/sirius#2181. Consumer bridge tracking: polardb/duckdb-paimon#139.

Need

A database embedding paimon-cpp needs to admit planning work before materializing an arbitrarily large native plan, and stop submitting work after cooperative cancellation. This is a feature request, not a claim that existing standalone planning is incorrect.

The integration pins 53f9c86d45aabb0a6f1a379271da07d7a9f27a3d. The following API gap also remains in inspected main 242decffdfdefef01dcd0e52e48c9a1e69792479:

  • ScanContextBuilder::WithMemoryPool routes pool-backed allocations, but does not bound ordinary STL planning containers or their transitive metadata allocations.
  • TableScan::CreatePlan() returns a complete plan; it exposes no planning cancellation/budget contract. SetLimit is a row-result pushdown, not a planning-memory quota: DataTableBatchScan::CreatePlan calls starting_scanner_->Scan before ApplyPushDownLimit.
  • FileStoreScan::ReadFileEntries builds a vector of futures for all supplied manifest metadata, then collects complete per-manifest entry vectors. A custom serial executor can prevent concurrent task execution, but does not itself bound those retained containers/results.

Source references at the inspected main:

Proposed Contract To Discuss

Provide optional query-scoped planning controls, preserving existing unlimited behavior by default. The API shape should be agreed with maintainers; a post-plan split-count check is not sufficient.

  • Check conservative manifest/entry/split/retained-metadata quotas before the corresponding allocation or task submission; identify which allocations are covered and what remains outside the guarantee.
  • Bound outstanding manifest work and retained results, including ordinary containers, rather than only pool-backed buffers.
  • Check a cooperative cancellation token before I/O/submission and inside substantial decode/merge loops. Do not promise interruption of an arbitrary blocking filesystem call unless the filesystem supports it.
  • Return a distinguishable resource-refusal/cancellation status, never partial success or EOF. Stop new submissions and drain accepted work before returning, including exceptions and submission failure.
  • Avoid changing process-global executors, Arrow thread capacity, or allocation resources.

Acceptance tests should use many small manifests and injected allocation/I/O/submission failures. Assert the limit is enforced before excess work is accepted, accepted jobs are drained on every terminal path, and cancellation cannot return a partial successful plan. Include serial/nested executor configurations to guard against deadlock. Keep row LIMIT semantics separate.

This issue does not request another Arrow batch-lifetime implementation; that work already exists in #259/#260. No hard whole-process-memory bound or prompt cancellation of uninterruptible I/O is claimed by the current integration.

Lingua principale
C++
Stelle
68
Fork
34
Merge medio
3g 18h
PR unite (30g)
44

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/paimon-cpp

Tutte le issue di apache/paimon-cpp

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.