State scorer collapses sibling `order_items` (and orders) — `_record_identity` keys child records by their parent id, causing false `task_completion = 0`

Open Beginner friendly
#36 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python
Domain
testing-qa

Research direction

Start in state_bench/scoring.py at _record_identity, then run the supplied reproduction for customer_support/13-cancel_partial. Verify that records with their own identifiers retain distinct keys, and that evaluate_state_requirements returns the expected score with no missing assertions; check the related 15-cancel_backordered case if needed.

Written by the indexing model from the issue text.

Description

Summary

evaluate_state_requirements reconstructs the final DB snapshot from the task-env
file, which stores entities as lists. scoring.py::_record_identity derives a
record's key from the first matching id field, but it checks owner/foreign keys
(order_id, customer_id, cart_id) before the record's own id (item_id,
…). An order_item carries both item_id and its parent order_id, so all line
items in the same order collapse under one key — only the last survives and its
siblings disappear from the reconstructed snapshot. Their state_requirements
then report as missing_assertions, failing the task even when the agent behaved
perfectly.

Environment

  • STATE-Bench v0.7.1 (commit a0ffc65)
  • state_bench/scoring.py, function _record_identity (around line 206)

Reproduction

Run from a STATE-Bench checkout (uses only shipped files — task
customer_support/13-cancel_partial):

import json
from pathlib import Path
from state_bench.scoring import _record_identity, evaluate_state_requirements
from state_bench.schemas import TaskDefinition, StateDiff

# Root cause: three sibling order_items collapse to one key.
items = [{"item_id": f"ITEM-{n}", "order_id": "ORD-1", "item_status": "confirmed"} for n in (1, 2, 3)]
print([_record_identity(i) for i in items])   # -> ['ORD-1', 'ORD-1', 'ORD-1']

# End-to-end: a correct partial cancellation of only the shirt (ITEM-9105).
task = TaskDefinition.from_dict(json.loads(
    Path("state_bench/domains/customer_support/tasks/13-cancel_partial.json").read_text()))
state_diff = StateDiff(modified={
    "orders": {"ORD-6016": {"status": {"old": "processing", "new": "partially_cancelled"}}},
    "order_items": {"ITEM-9105": {
        "item_status": {"old": "confirmed", "new": "cancelled"},
        "refund_amount": {"old": None, "new": 59},
        "refund_method": {"old": None, "new": "credit_card"},
    }},
})
res = evaluate_state_requirements(task, state_diff)
print(res.score, res.reasoning, res.details)

Actual output (v0.7.1)

['ORD-1', 'ORD-1', 'ORD-1']
0 State requirements failed: missing_assertions=2 {'missing_assertions': [
  {'entity_type': 'order_items', 'record_key': 'ITEM-9106', 'field': 'item_status', 'value': 'confirmed'},
  {'entity_type': 'order_items', 'record_key': 'ITEM-9107', 'field': 'item_status', 'value': 'confirmed'}]}

ITEM-9106 and ITEM-9107 are never modified by the agent and should remain
confirmed, but they vanish from the reconstructed snapshot because all three
items collapsed under ORD-6016.

Expected output

['ITEM-1', 'ITEM-2', 'ITEM-3']
1 All required state assertions matched the saved state_diff. None

Impact

Any task asserting state on a sibling line item that is not the last in its order
gets phantom missing_assertions and a false task_completion = 0. Observed on
at least customer_support/13-cancel_partial and 15-cancel_backordered. The
same class affects orders keyed by customer_id for customers with multiple
orders (currently masked when the task modifies the order, so the diff re-adds the
correct key).

Suggested fix

Order _record_identity so a record's own id precedes owner/foreign keys.
Records without their own id (orders, customers, products) keep their previous
keying; only records that have both their own id and a parent id change:

 def _record_identity(record: dict[str, Any], fallback_index: int | None = None) -> str | None:
     for key in (
         "cart_item_id",
-        "cart_id",
-        "customer_id",
-        "order_id",
         "item_id",
         "booking_id",
         "reservation_id",
         "rental_id",
+        "cart_id",
+        "order_id",
+        "customer_id",
         "id",
         "product_id",
     ):
Dominant language
Python
Stars
91
Forks
13
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/STATE-Bench

All issues in microsoft/STATE-Bench

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.