Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

The chunks of one document are extracted one at a time

Open
  • Is something wrong?
#588 13 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@jeremycapps is already working on this.

Since Sep 25, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
68/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
backend, performance

Research direction

Start at extraction.rs and the quoted for chunk in chunks.iter() loop: spawn the per-chunk model calls bounded by the model_limits gate, but apply replies to resolution strictly in chunk order. Read the surrounding resolution path (handled_by_name, response_claims, the ledger write) plus how unextracted failures and gate backoff are reported today, since those must not change. Done = three documents uploaded together finish near chunks / gate calls, with ledger and fact/entity counts within #583's measured variance.

Written by the indexing model from the issue text.

Description

enhancement

One document is one job, and inside the job the chunks are extracted one after another (extraction.rs, for chunk in chunks.iter()). A model call takes about 55 seconds on DeepSeek-V3 with the schema.org candidates, so a document's extraction time is chunks × 55 s no matter what the deployment's worker_concurrency or the model's max_concurrent say: those only let more documents run at once.

Measured on the #582 runs, three documents uploaded together: anthropic.txt (27 chunks) 25 min, removal-of-sam-altman-from-openai.txt (23) 21 min, openai.txt (73) 68 min. The two small ones finish and the base then waits three quarters of an hour on one document with two idle workers, while the model gate (10 concurrent by default) sits at one call.

What should happen

The chunks of one document are extracted concurrently, bounded by the model's concurrency gate (model_limits, the same one every other call goes through), and the results are applied in chunk order so that entity resolution and the ledger behave exactly as they do today: resolution is order-sensitive (a later chunk's namesake logic reads what earlier chunks bound, handled_by_name, response_claims), so the model calls are the parallel part and the resolution stays sequential over the replies in sequence order.

Expected effect on the same three documents: bounded by the gate, not by the longest document. With a gate of 10, openai.txt becomes roughly 73 / 10 × 55 s ≈ 7 min of calls plus resolution.

Not this

  • Not a bigger worker_concurrency: it changes nothing for a single document.
  • Not parallel resolution: the order-dependent parts stay in order.
  • Not a change to what is extracted: the same prompt, the same replies, the same ledger; a run before and after on the same documents should differ only in wall-clock time (and the run-to-run variance the model already has).

Acceptance

  • Three documents uploaded together finish in about the time of the longest document's chunks / gate calls, not chunks calls.
  • A failed chunk is reported the same way it is today (unextracted, the per-document summary), and a rate-limited call still backs off through the gate rather than retrying in a tight loop.
  • The extraction ledger and the fact/entity counts on a re-extraction are within the run-to-run variance already measured (#583's table).

Related: #526 (a worker held while the ontology index is built, the other reason a document waits), #583 (the runs that measured this).

Status 2026-10-22

Done: cut 1 (#980), the bench knob and tables (#1002, #1005, #1014), the known measurement, and the decision of 2026-10-01 in the comments.

Decided design (not yet built): chunk 1 runs alone and seeds the known list; the remaining chunks run at once under the model gate with chunk 1's list; apply stays sequential in chunk order; a failed chunk is reported as unextracted, as today. The loop in extraction_open.rs is still for chunk in chunks.iter(), and UTOPIA_EXTRACT_KNOWN_IN_PROMPT only changes the prompt.

  • Cut 2: the seed-then-parallel loop
Dominant language
Rust
Stars
8.1k
Forks
1.1k
Avg merge
21h 2m
Merged PRs (30d)
263

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deeplethe/utopia

All issues in deeplethe/utopia

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.