core: output order changes between builds
Maintainer antworten meist innerhalb von 3 Tagen
Bewertung
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Anfängerfreundlichkeit
- 76/100
- Issue-Typ
- Bug
- Klarheit
- Klar beschrieben
- Aktivitätsstatus
- Aktiv
- Tech-Stack
- javascript, node.js
- Bereich
- build-system, tooling
Rechercherichtung
Start with packages/core/src/threading/parallel.mjs and the existing test in packages/core/src/threading/__tests__/parallel.test.mjs; use the fake pool described in the issue to confirm output order. Then inspect the globSync calls in packages/core/src/generators/ast/generate.mjs and ast-js/generate.mjs, checking the status of PRs #1180 and #1179 first. Add focused tests and verify repeated builds produce stable output order.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Building the same docs twice can give different files, because the order of the output depends on which worker finishes first. Two things combine:
createParallelWorker().stream()sends every chunk to the worker pool at once, then yields the results withPromise.race, so in whatever order the workers happen to finish.- The
astandast-jsgenerators list input files withtinyglobby'sglobSync, which doesn't sort, so even the order chunks are sent in can change between machines.
Generators that don't sort for themselves pass that order on. These files can change between identical builds:
sitemap.xml(<url>order)llms.txtandllms-full.txt(link and page order)orama-db.json(document order; its ids are a separate issue)- the legacy
all.html(table of contents, sidebar and content) api-docs.jsonfromjson-simpleapilinks.json(key order)
That makes diffs noisy, wastes caching, and keeps CI from comparing outputs byte for byte (today it can only compare file sizes). Generators that already sort need no change: legacy-json-all, json-all, the React all.html, jsx-ast and legacy-html.
Where to look
packages/core/src/threading/parallel.mjs:99-137: every chunk is submitted at once, then thewhile (pending.size)loop yields withawait Promise.race(pending)(line 126).packages/core/src/caching.mjs:21-24:collectAsyncGeneratorkeeps chunks in the order they arrive.packages/core/src/generators/ast/generate.mjs:117-124andast-js/generate.mjs:47-49: unsortedglobSyncresults.packages/node-legacy/src/legacy-json-all/generate.mjs:39-52: an example of a consumer that already sorts.
How to reproduce
You don't need real workers. With a fake pool where later chunks finish first:
import createParallelWorker from './packages/core/src/threading/parallel.mjs';
const pool = {
run: async task => {
await new Promise(resolve => setTimeout(resolve, 50 - task.input[0] * 10));
return task.input;
},
};
const worker = createParallelWorker('x', { name: 'x' }, pool, {
threads: 2,
chunkSize: 1,
});
const out = [];
for await (const chunk of worker.stream([0, 1, 2, 3, 4], {})) out.push(...chunk);
console.log(out); // [ 4, 3, 2, 1, 0 ], expected [ 0, 1, 2, 3, 4 ]
(Run it from the repo root as an .mjs file. If the logger import gets in the way, the test file below is the easier place to try it.)
Suggested approach
-
Yield in submission order. All chunks still start at once, so it isn't any slower; the loop just awaits them in order:
const promises = chunks.map(indices => pool.run(createTask(/* ... */))); for (const [index, promise] of promises.entries()) { const result = await promise; // keep the "Chunk N/M completed" debug log yield result; }The
{ promise, result }wrapper and thependingSet can go, and the JSDoc should say "in submission order". -
Sort the glob results with a plain
.sort(), notlocaleCompare, so the order doesn't depend on the machine's locale. Inast/generate.mjs, sort each pattern's results before mapping them to[child, parent], which keeps the order ofconfig.input.
Two open PRs touch the same code, so check where they stand when you start:
- #1180 replaces
tinyglobbywithfindFilesinpackages/core/src/utils/glob.mjs. If it has landed, sort there instead, once, for every caller. - #1179 already makes the single-thread path run chunks one after another, in order. If it has landed, only the worker pool path needs the change.
How to test
In packages/core/src/threading/__tests__/parallel.test.mjs, add a case with a fake pool like the one above (no real workers needed) and assert that [0, 1, 2, 3, 4] comes back in order. For the glob sort, packages/core/src/generators/ast/__tests__/generate.test.mjs already creates a temp folder: write a few files in a jumbled order and assert the generated paths come back sorted. Run everything with node --run test.
Things to watch out for
- Fix the order once, at the source. Don't add sorting to each consumer (
sitemap,llms-txt, ...). - In
ast/generate.mjs, sort before the.mapinto[child, parent]tuples, because.sort()on arrays would compare them as strings. - After this, the legacy
all.htmlfollowslegacy-html's order (modules sorted by name). Whether it should followconfig.indexlikelegacy-json-allis a separate question. orama-db.jsonalso needs stable document ids to be reproducible: #1184.
Getting started
New to doc-kit? CONTRIBUTING.md covers setup, running the tests and adding a changeset (node --run changeset, a patch for @doc-kit/core). Leave a comment here if you'd like to take it on.
- Vorherrschende Sprache
- JavaScript
- Sterne
- 70
- Forks
- 75
- Ø Merge
- 2 T. 11 Std.
- Gemergte PRs (30 T.)
- 50
Entwicklungsumgebung
- Kein Dockerfile und keine Docker-Compose-Datei
- Hat eine Pull-Request-Vorlage
- Beitragsleitfaden lesen
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus nodejs/doc-kit
-
Schwierigkeit 2/5 Ein halber Tag Anfängerfreundlichkeit 68/100
nodejs/doc-kit#1085 · 6 Kommentare ·
Maintainer antworten meist innerhalb von 3 Tagen
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 64/100
nodejs/doc-kit#1054 · 1 Kommentar ·
Maintainer antworten meist innerhalb von 3 Tagen
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 55/100
nodejs/doc-kit#1194 · 1 Reaktion ·
Maintainer antworten meist innerhalb von 3 Tagen
-
core: incremental generation against an existing output (`--incremental`)Evtl. vergeben @avivkeller hat das vor 1 Tag übernommen. OffenCLI
nodejs/doc-kit#1189 · 2 Kommentare · 1 zugewiesene Person ·
Maintainer antworten meist innerhalb von 3 Tagen
-
cli: add a `dev` command with in-memory builds and HMREvtl. vergeben @avivkeller hat das vor 1 Tag übernommen. OffenCLI Web Generator
nodejs/doc-kit#1188 · 1 Reaktion · 1 zugewiesene Person ·
Maintainer antworten meist innerhalb von 3 Tagen
Ähnliche Issues
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 88/100
micromatch/picomatch#243 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
Maintainer antworten meist innerhalb von 1 Tag
-
enhancement
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 70/100
b310-digital/teammapper#1515 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 64/100
c1overzz/dsh-bili-scroll#153 ·
Maintainer antworten meist innerhalb von 1 Tag
-
01 type: bug 21 status: confirmed 41 lvl: easy 62 scope: viewer
Schwierigkeit 1/5 Unter einer Stunde Anfängerfreundlichkeit 90/100
Maintainer antworten meist innerhalb von 1 Tag