Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Feature] Add query-scoped planning limits and cooperative cancellation

Aberta
  • Algo errado?
#435 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
8/100
Tipo de issue
Funcionalidade
Clareza
Precisa de esclarecimento
Status de atividade
Ativa
Stack de tecnologia
cpp
Domínio
data-engineering

Direção de pesquisa

This is design work, so start by reading the API surfaces in include/paimon/scan_context.h and include/paimon/table/source/table_scan.h, then the limit ordering in src/paimon/core/table/source/data_table_batch_scan.cpp and the manifest fan-out in src/paimon/core/operation/file_store_scan.cpp (ReadFileEntries). The first step is agreeing the API shape with maintainers on this thread. Done means an approved contract plus tests that inject allocation, I/O and submission failures.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Parent integration: sirius-db/sirius#2181. Consumer bridge tracking: polardb/duckdb-paimon#139.

Need

A database embedding paimon-cpp needs to admit planning work before materializing an arbitrarily large native plan, and stop submitting work after cooperative cancellation. This is a feature request, not a claim that existing standalone planning is incorrect.

The integration pins 53f9c86d45aabb0a6f1a379271da07d7a9f27a3d. The following API gap also remains in inspected main 242decffdfdefef01dcd0e52e48c9a1e69792479:

  • ScanContextBuilder::WithMemoryPool routes pool-backed allocations, but does not bound ordinary STL planning containers or their transitive metadata allocations.
  • TableScan::CreatePlan() returns a complete plan; it exposes no planning cancellation/budget contract. SetLimit is a row-result pushdown, not a planning-memory quota: DataTableBatchScan::CreatePlan calls starting_scanner_->Scan before ApplyPushDownLimit.
  • FileStoreScan::ReadFileEntries builds a vector of futures for all supplied manifest metadata, then collects complete per-manifest entry vectors. A custom serial executor can prevent concurrent task execution, but does not itself bound those retained containers/results.

Source references at the inspected main:

Proposed Contract To Discuss

Provide optional query-scoped planning controls, preserving existing unlimited behavior by default. The API shape should be agreed with maintainers; a post-plan split-count check is not sufficient.

  • Check conservative manifest/entry/split/retained-metadata quotas before the corresponding allocation or task submission; identify which allocations are covered and what remains outside the guarantee.
  • Bound outstanding manifest work and retained results, including ordinary containers, rather than only pool-backed buffers.
  • Check a cooperative cancellation token before I/O/submission and inside substantial decode/merge loops. Do not promise interruption of an arbitrary blocking filesystem call unless the filesystem supports it.
  • Return a distinguishable resource-refusal/cancellation status, never partial success or EOF. Stop new submissions and drain accepted work before returning, including exceptions and submission failure.
  • Avoid changing process-global executors, Arrow thread capacity, or allocation resources.

Acceptance tests should use many small manifests and injected allocation/I/O/submission failures. Assert the limit is enforced before excess work is accepted, accepted jobs are drained on every terminal path, and cancellation cannot return a partial successful plan. Include serial/nested executor configurations to guard against deadlock. Keep row LIMIT semantics separate.

This issue does not request another Arrow batch-lifetime implementation; that work already exists in #259/#260. No hard whole-process-memory bound or prompt cancellation of uninterruptible I/O is claimed by the current integration.

Linguagem predominante
C++
Estrelas
68
Forks
34
Merge médio
3d 18h
PRs com merge (30d)
44

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/paimon-cpp

Todas as issues de apache/paimon-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.