Flaky CI: intermittent tests-without-auth failures (boost timeDecay ~1-in-6, plus two confirmed serialization bugs)

Aperta
#470 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
docker, node.js, typescript

Direzione di ricerca

Inizia da test/collections/query/integration.test.ts e dai test falliti descritti nell’issue, quindi esamina src/collections/serialize/index.ts, src/collections/deserialize/index.ts e test/collections/journey.test.ts. Il lavoro è completo quando il boost test è deterministico, decay raggiunge il proto generato, l’output di debug è stato rimosso, UUIDs mancanti e un setup fallito non causano errori di Cleanup opachi e CI non presenta più flakiness per questi casi.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

tests-without-auth fails intermittently across all Weaviate versions and both Node versions, on unrelated PRs. Re-running the identical commit turns it green, so the code under test is not the cause.

Most recent evidence: run 33513846023 failed, then passed on all 8 jobs after a re-run with no code change.

At least three distinct flakes are involved. One is now understood in detail; the other two are not.


1. query > boost > 'timeDecay' — understood, ~1-in-6 failure rate

test/collections/query/integration.test.ts

AssertionError: expected [ …(3) ] to not deeply equal [ …(3) ]

The assertion is:

const baseline = ids(await collection.query.nearVector([1, 2, 3]));
const boosted  = ids(await collection.query.nearVector([1, 2, 3], { boost }));
expect(boosted).not.toEqual(baseline);
Measured behaviour

Reproduced locally against semitechnologies/weaviate:1.38.2, rebuilding the collection each trial:

  • 5/30 trials (16.7%) failed on unmodified main.
  • Across rebuilds the boosted ordering spans all six permutations of the three objects, near-uniformly. 1/6 = 16.7%, matching the observed rate.
  • Within a single collection, both baseline and boosted are perfectly stable (0/30 unstable). This is why re-running the same query never reproduces it — only rebuilding the collection does.
Ruled out
Hypothesis Result
All three createdAt set to the same Date.now() ruled out — spreading them failed at the same rate
blue/green share vector [1,2,3] (distance tie) ruled out — distinct vectors failed at the same rate
Query racing vector-index build after insertMany ruled out — 250ms and 1000ms waits changed nothing
Assessment

The test's premise is the problem: it assumes a boost must reorder three near-tied results, which the server does not guarantee. Retuning the input values will not hold — 467e5a7b already attempted exactly that in July, and this is the same test failing again.

Suggested direction: assert something deterministic (an expected ordering over well-separated data, or scores/membership) rather than "the order changed". The other three boost cases in the same it.each (filter, numericDecay, numericProperty) have never flaked.


2. Two confirmed bugs found while investigating

These are independent of the flake and are separately actionable.

2a. decay is silently dropped from Boost.timeDecay / Boost.numericDecay

The public type uses decay; the generated proto field is decayValue:

  • src/collections/types/query.ts:211,223 — decay?: number
  • src/proto/v1/search_get.ts:351-367 — decayValue?: number
  • src/collections/serialize/index.ts:1645,1659 — fromPartial({ ...timeDecay, curve })

fromPartial discards the unknown decay key, so the value never reaches the wire. Observed payload:

{"timeDecay":{"property":"createdAt","origin":"2024-01-01T00:00:00Z","scale":"35d","curve":3}}

With a two-line mapping (decayValue: decay) it becomes:

{"timeDecay":{"property":"createdAt","origin":"...","scale":"35d","curve":3,"decayValue":0.2}}

Any user passing decay today has it ignored, with no error. Note: applying this fix did not make the ordering deterministic, so it is not the cause of the flake above.

2b. Debug logging shipped in the client

src/collections/serialize/index.ts:1679

console.error(JSON.stringify(out));

Inside Serialize.boostGRPC — writes to stderr on every boosted query.


3. Not yet understood

3a. data > dryRun deleteMany — TypeError: Stringified UUID is invalid
❯ stringify node_modules/uuid/dist-node/stringify.js:31:15
❯ src/collections/deserialize/index.ts:536:19   id: stringify(obj.uuid),

The server returns a BatchDeleteObject with no uuid; the proto defaults it to new Uint8Array(0) (src/proto/v1/batch_delete.ts:303,332), unsafeStringify then indexes byteToHex[undefined] and produces an invalid string. Not reproduced locally; why the server omits the field is unresolved. Regardless, the call site is unguarded and turns this into an opaque TypeError.

3b. collections/journey.test.ts — fetch failed in beforeAll
WeaviateStartUpError: Weaviate failed to startup with message: fetch failed   (src/connection/helpers.ts:123)

This suite connects to a fixed external WCD cluster rather than the CI container, so all seven matrix jobs hit the same shared cluster concurrently and a transient network failure kills the suite (observed with zero failed assertions: 1204 passed | 83 skipped).

Separately, afterAll at test/collections/journey.test.ts:16 is unguarded:

afterAll(() => client.collections.delete(collectionName));

When beforeAll fails, client is undefined and this throws a second, misleading TypeError: Cannot read properties of undefined (reading 'collections'), obscuring the real cause. A client?. guard would fix that much.

Worth considering whether a shared, externally-mutable cluster belongs in the blocking CI path at all — an upgrade to it recently reddened every open PR simultaneously (fixed in #469).

Lingua principale
TypeScript
Stelle
101
Fork
43
Merge medio
7h 31m
PR unite (30g)
6

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di weaviate/typescript-client

Tutte le issue di weaviate/typescript-client

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.