`closed_q_a.yaml` drops OpenAI’s scoping clause; `factuality.yaml` has a traceable typo

Start by reading closed_q_a.yaml and factuality.yaml, then compare their prompt wording with conformance/corpus/braintrust-autoevals__closedqa.json and conformance/corpus/braintrust-autoevals__factuality.json. Restore the identified scoping clause and typo, and determine whether the factuality score ordering is intentional or needs a separate change; done means the template text and scoring behavior match the accepted specification.

I’ve published a conformance census of the default judge templates across nine eval frameworks: https://github.com/idilgozel/evaluator-integrity

I found two things in autoevals that I think are worth flagging.

closed_q_a.yaml drops a scoping clause.OpenAI’s original template says the reasoning instruction applies to “your reasoning about the criterion.” The autoevals version drops that part. The resulting COT_SUFFIX therefore gives the model an unscoped instruction to reason step by step, and this is appended in code.

Record: conformance/corpus/braintrust-autoevals__closedqa.json

factuality.yaml has “in a step-by-step matter”The original OpenAI wording is “in a step by step manner.” The autoevals version has “in a step-by-step matter.”*This looks like a small mutation of the same template, so it may also be useful as a lineage fingerprint when comparing where these templates came from.

There is also a scoring issue I think is worth looking at. E (“differences don’t matter”) gets 1.0 without defining what counts as a difference that doesn’t matter. Also, B (correct answer + unverified extra claims) gets 0.6, while A (correct but incomplete) gets 0.4. So the default scoring gives a higher score to an answer containing unchecked additional claims than to one that is simply incomplete.

Record: conformance/corpus/braintrust-autoevals__factuality.json

I’m happy to open a PR for the scoping clause and typo if useful. I didn’t want to open one without checking first.