[CI Flaky] test_other_bits 2-bit UT fails intermittently (Accuracy 0.18 < threshold 0.195)

Open
  • Is something wrong?
#2,550 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@luoyu-intel is already working on this.

Since Oct 29, 2026.

Assessment

This issue has not been assessed yet.

Description

bug CI-known-issue

Bug Description

TestAutoRoundARKBackend::test_other_bits[cpu-dtype0-2-32-False-auto_round] (the only 2-bit / w2g32 case in the ARK UT suite) fails intermittently in CI with:

E   AssertionError: Accuracy 0.18 is below threshold 0.195
FAILED unit/test_ark/test_model.py::TestAutoRoundARKBackend::test_other_bits[cpu-dtype0-2-32-False-auto_round]

Same code, same runner — sometimes passes, sometimes fails. This is a flaky/borderline threshold issue, not a real accuracy regression.

Root Cause Analysis

Evidence from the failing CI log (Azure DevOps build 78323, log 58):

  1. 2-bit quantization on opt-125m is inherently near-random quality.
    The quantized model generates garbage text (Prompt: "Hello,my name is" -> Generated: " Mys. I am"), consistent with a lambada accuracy of ~0.18 — barely above random chance.

  2. The threshold 0.195 is calibrated for 4/8-bit, not 2-bit.
    All other cases (formats, awq_fp16) run at 4/8-bit and pass; the 2-bit case is the only one sitting at the edge of the noise band.

  3. Evaluation uses only ~100 lambada_openai samples (limit), with no fixed seed.
    Binomial standard error ≈ sqrt(0.19 * 0.81 / 100) ≈ ±0.04 (2σ ≈ ±0.08) — far wider than the 0.015 gap between measured 0.18 and threshold 0.195. Run-to-run eval-subset sampling (plus CPU quantization nondeterminism) swings the measured accuracy across the threshold, so pass/fail is essentially a coin flip.

Conclusion: The test design is at fault — a hardcoded 0.195 threshold applied to a 2-bit model measured on a tiny unseeded sample set. It will keep flaking regardless of code changes.

Suggested Fixes (pick one)

  • Preferred: For the 2-bit case, drop the accuracy assertion (or use a separate relaxed threshold) and only verify quantization + repack + inference succeed. 2-bit has no accuracy-parity requirement in this test.
  • If accuracy must be asserted: raise limit (e.g. 100 → 500, dropping SE to ~±0.02) and fix the random seed so the measurement is deterministic, then set the threshold against a measured 2-bit baseline with margin.

Environment

Error Output

helpers.py:139: in evaluate_accuracy
    assert acc > threshold, f"Accuracy {acc} is below threshold {threshold}"
E   AssertionError: Accuracy 0.18 is below threshold 0.195
FAILED unit/test_ark/test_model.py::TestAutoRoundARKBackend::test_other_bits[cpu-dtype0-2-32-False-auto_round] - AssertionError: Accuracy 0.18 is below threshold 0.195
Dominant language
Python
Stars
1.7k
Forks
195
Avg merge
2d 11h
Merged PRs (30d)
59

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from intel/auto-round

All issues in intel/auto-round

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.