[Bug]: cua-s1-forms answers a confident skip outside its 55-concept label catalogue (97.5% in-distribution vs 29.3% held-out)

Start with libs/cua-s1/MODEL_CARD.md, especially its Limitations section, and inspect cua_s1/concepts.py for the 55-entry vocabulary. Document the observed out-of-catalogue behavior and its high-confidence skip output, then check the model-card comparison references, including docs/RESULTS.md. Done when the operating boundary is actionable for downstream integrators and the comparison treatment matches the project's decision.

Primary area

Documentation

Summary

On a held-out set whose field concepts fall outside the 55-entry catalogue in cua_s1/concepts.py, cua-s1-forms scores 29.3% (12/41) while answering skip on 36 of 41 elements at 0.974 mean confidence. The same checkpoint scores 97.5% on in-distribution episodes from the repository's own cua_s1.synth. The model card's Limitations section says the checkpoint is "not validated on arbitrary real-world forms outside the demo set", but does not state that the failure outside the catalogue is a confident skip rather than a low-confidence or abstaining output, which is the part that matters for a downstream executor.

Reproduction

1. Convert the published .pt (see #3977) to ckpt/model.safetensors + ckpt/config.json.
2. Run the scripts in https://gist.github.com/0xsline/2d492b5baa5b18110b97e733f0bb697d:

PYTHONPATH=src:. .venv/bin/python device_matrix.py   # device/batching control
PYTHONPATH=src:. .venv/bin/python ablate.py          # in-distribution baseline, option-count, vocabulary
PYTHONPATH=src:. .venv/bin/python ablate2.py         # 2x2x2 over doc label / form label / form title

heldout.py contains 41 decisions over 5 forms (logistics, hardware, DevOps, veterinary, and one Chinese form). Contexts are rendered with the package's own cua_s1.schema.render_context, and options with Entity.option(), so the input format is identical to training. The action mix is 25 fill, 7 skip, 4 check, 5 click.

Expected behavior

The model card documents the operating boundary in a way a downstream integrator can act on: which label vocabulary the checkpoint depends on, and what it does when an element falls outside it.

Actual behavior

in-distribution (cua_s1.synth, seeds 900000-900119, 3070 decisions) : 2993/3070 = 97.49%
  per action: skip 1632/1638 · fill 1137/1208 · click 120/120 · check 104/104

held-out label vocabulary (41 decisions)                            :   12/41 = 29.27%
  per action: fill 2/25 · skip 7/7 · check 2/4 · click 1/5
  36 of 41 predictions are `skip`; mean confidence 0.974

skip scores 7/7 only because the model emits skip almost everywhere. check and click also fail on held-out forms, for example Button "Register Device" and CheckBox "Register for firmware updates", even though those decisions do not depend on any document entity.

Environment

Cua component and version/commit: libs/cua-s1 @ b7f7e2d (PR #3964), artifact cua-ai/cua-s1-forms @ 2026-09-18T19:35:43Z
Operating system and architecture: macOS 27.0.0, arm64 (Apple M5)
Python / torch: CPython 3.13.2, torch 2.14.0, safetensors 0.8.0

Evidence

Three controls rule out the obvious harness explanations.

Device and batching. Identical predictions across all four combinations, so this is not an MPS or padding artifact:

cpu/one batch  29.3%   mps/one batch  29.3%
cpu/batch=1    29.3%   mps/batch=1    29.3%      41/41 identical predictions

Harness correctness. The same code path reaches 97.49% on cua_s1.synth episodes, so the context and option rendering are right.

Option count. The held-out forms expose 6 entities + 3 fixed actions, against a mean of 20.4 options in-distribution. Trimming in-distribution option lists to the same width does not move accuracy, so width is not the cause:

full option lists (~20 options) : 2993/3070 = 97.49%
trimmed to 6 entities + 3       : 2998/3070 = 97.65%

What does move it is the label wording. Holding structure, shape, values, and action mix fixed and only renaming labels to phrases from concepts.py:

held-out labels (logistics/DevOps/veterinary/Chinese) : 12/41 = 29%
same shape, in-catalogue labels                       : 11/11 = 100%

A 2x2x2 over the three text surfaces isolates which one carries the signal:

doc labels        form labels       title     accuracy
in-catalogue      in-catalogue      in        8/8 = 100%
in-catalogue      in-catalogue      out       8/8 = 100%
in-catalogue      out               in        2/8 =  25%
in-catalogue      out               out       2/8 =  25%
out               in-catalogue      in        4/8 =  50%
out               in-catalogue      out       5/8 =  62%
out               out               in        2/8 =  25%
out               out               out       2/8 =  25%

The form title is irrelevant. The element (form) label dominates: replacing it with unseen wording drops accuracy to 25% even when every document label is in-catalogue. Document labels contribute less but still matter.

Workaround and additional context

I am not disputing the reported in-distribution numbers; I reproduced that regime at 97.5% with arbitrary seeds. The gap I want to record is the shape of the failure, not its existence, because the model card already anticipates OOD weakness in prose.

Two suggestions, both documentation-only:

1. State in the model card that the checkpoint keys on the element label matching the 55-concept form_labels vocabulary, and that outside it the output is a high-confidence skip rather than an abstention. An executor that treats skip as "nothing to do here" will silently no-op an entire form instead of surfacing that the form is out of scope. MODEL_CARD.md already asks integrators not to treat apparent task completion as proof of correctness; this is a concrete instance worth naming.
2. Consider whether the model card's headline comparison belongs there at all. #3962's accepted decision lists "publishing ... hosted-provider comparison artifacts" and "advertising checkpoint accuracy or platform support without independently reproducible evidence" as rejected for this implementation, and the docs/RESULTS.md that the comparison points to is missing (#3977). For what it is worth, on my held-out set hosted jev-latest with a one-sentence prompt scores 37/41 = 90.2% on the same items, which is the opposite ordering from the card. That number is not comparable to the card's 83.6%, since the task framing and item set differ, which is precisely why I think the comparison needs a reproducible harness next to it or no place on the card.

Happy to open a documentation pull request against libs/cua-s1/MODEL_CARD.md for point 1, or to contribute the held-out set under libs/cua-s1/evals/ if an out-of-catalogue regression set is useful to you.

Submission checks

• I searched for duplicate issues and active pull requests.
• This issue describes a single problem.
• I removed credentials, private data, sensitive screenshots, and vulnerability details from this public report.