Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Feature] Cut metadata round trips when planning a scan (known manifest sizes, listing probes, parallel manifest-list reads)

Fechada
  • Algo errado?
#337 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
58/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Domínio
data-engineering

Direção de pesquisa

Comece por ObjectsFile::Read, ReadIfFileExist, ReadArrowBatches e ReadFileSegment; em seguida, rastreie os callers de ManifestFile, ManifestList, FileStoreScan e SnapshotFileCollector. Inspecione também FileUtils::ListVersionedFileStatus e JindoFileSystem::ListDir. Está concluído quando tamanhos conhecidos evitarem chamadas de status de metadados, as listagens de diretórios inexistentes continuarem funcionando e as leituras dos manifestos base/delta preservarem a ordem enquanto são executadas de forma concorrente.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Scan planning pays several object-store round trips that are avoidable, either because the answer is already in metadata it has read or because one call can answer what two are asking.

  • Manifest reads re-resolve a length the metadata already carries. A manifest list records each manifest's fileSize, and a snapshot records its base/delta/changelog manifest-list sizes. But ObjectsFile::Read* opened these files with a bare Open(path), so on a remote store every manifest and manifest list paid a getFileStatus/HeadObject round trip just to learn a length planning already had. This is the metadata-path counterpart of the data-file fast path in Open(const FileStatus&).
  • ListVersionedFileStatus probes existence before listing. It called Exists(dir) then ListDir(dir). Every file system already lists a missing directory as an empty result rather than an error, so the probe only decided whether to make a call that answers the same question — an extra round trip on every schema/snapshot/versioned-file lookup.
  • Jindo ListDir asks the store twice. It called Exists(dir) then GetFileStatus(dir); a single GetFileStatus answers both "is it there" and "is it a directory".
  • ScanMode::ALL reads the base and delta manifest lists serially. They are two independent files and neither read depends on the other, so the two metadata round trips are paid one after the other instead of together.

For scans over many manifests, or against a high-latency object store, these round trips are a measurable and entirely avoidable part of planning latency.

Solution

  • Thread an optional known length through the metadata read path: ObjectsFile::Read/ReadIfFileExist/ReadArrowBatches/ReadFileSegment gain a std::optional<int64_t> file_size (default std::nullopt), and a new OpenForRead helper opens with Open(FileStatus(path, size)) when the length is known and falls back to Open(path) when it is not. ManifestFile::ReadBucketEntries forwards it, ManifestList::ReadBase/Delta/ChangelogManifests pass the sizes recorded on the snapshot, and FileStoreScan / SnapshotFileCollector pass each ManifestFileMeta::FileSize().
  • Drop the Exists() probe in FileUtils::ListVersionedFileStatus and list directly.
  • Collapse Jindo ListDir to a single GetFileStatus, mapping the SDK not-found to an empty listing (as the other file systems do) and propagating any other error.
  • In FileStoreScan::ReadManifestsWithSnapshot, read the base and delta manifest lists concurrently through the existing executor_ (Via + CollectAll), preserving base-then-delta order.

The size fields stay optional, so a snapshot or manifest list written before they existed keeps reading through the Open(path) fallback — this is an optimization, not a new requirement on the metadata.

Anything else?

  • The length handed to Open(FileStatus) is trusted, not re-validated, so a stale or wrong size would surface as a short or failed read. Manifest and manifest-list files are write-once and never rewritten, and the size is recorded by the same commit that wrote the file, so it cannot go stale underneath the read; a negative length is rejected by the base Open(const FileStatus&).
  • The size fast path benefits stores that override Open(const FileStatus&) (today ObjectStoreFileSystem; ResolvingFileSystem forwards it). JindoFileSystem currently only overrides Open(path), so for Jindo the manifest-size threading is inert until it gains that override; the Jindo win here is the single-call ListDir.
  • No storage format or protocol change, and no new public API: the changed signatures are internal src/paimon helpers and the new parameter is defaulted.

Are you willing to submit a PR?

  • I'm willing to submit a PR!
Linguagem predominante
C++
Estrelas
65
Forks
29
Merge médio
2d 6h
PRs com merge (30d)
71

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/paimon-cpp

Todas as issues de apache/paimon-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.