determinism violation prints only a 2,000-character prefix, hiding the first difference

Open Beginner friendly
#81 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
64/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
javascript
Domain
testing

Research direction

Search the oracle-gaps harness for the code that formats the oracle-gaps.report violation message and the 2,000-character cut on each recorded text. Reproduce with vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "an edit that changed the cold result after a hang replaced the only context", then use that log to confirm the printed prefix hides the divergence. Done means the message reports the first differing offset in each text with roughly 200 characters of context, plus each observation's kind and the two lengths, with nothing truncated before the offset.

Written by the indexing model from the issue text.

Description

Summary

When two runs of a constructed history record different observations, the determinism violation prints the
observation's text only as a prefix. The first 2,000 characters are identical, and the output stops there, so the
difference is not shown. A diagnosis then needs a local patch to see where the two traces diverge.

The violation should print the first differing offset in each recorded text, with context on both sides, and the
observation's kinds and lengths, so the failure can be read without a patch.

Example

tests/gap-report-oracle.property.test.mjs, test "constructed history: an edit that changed the cold result after a
hang replaced the only context" (run with -t), at commit 11105c5 on fix/gap-cli-retention (#61):

Error: oracle-gaps.report: two runs of "constructed history: an edit that changed the cold result after a hang replaced
the only context" recorded different traces at observation 0 ("cli").
First run:  cli {"lead":[...
Second run: cli {"lead":[...

Both texts are the same for 1,996 characters, and the framework cuts them at 2,000. The difference is not visible.

Log: /tmp/oom/h2h-control.log (on the reporting machine). Replay: vitest --run --config oracle-gaps/vitest.config.mjs gap-report-oracle -t "an edit that changed the cold result after a hang replaced the only context".

Why it matters

The determinism check is how a timing dependency is caught. A check that cannot say what differed cannot tell whether
the difference is a wall-clock cut, a reordering, or a different gap, so the failure cannot be classified.

Expected

  • The first differing offset in each text, with about 200 characters of context on each side.
  • The observation's kind and the lengths of the two texts.
  • Nothing cut before the offset.

Limits

This is a diagnostic issue. It doesn't say whether the underlying nondeterminism is a bug (the wall-clock cut issue may
be the cause, tracked separately).

Dominant language
JavaScript
Stars
5
Forks
1
Avg merge
6h
Merged PRs (30d)
56

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from TanStack/oracles

All issues in TanStack/oracles

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.