Agent Learning Track compliance: deterministic use of retrieved learnings before domain-tool execution
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
Research direction
Start by reviewing the custom BaseLLMClient and BaseAgent references, datasets/train_task_trajectories/, issue #26, and the STATE-Bench harness behavior described here. The work is complete when maintainers clarify whether deterministic middleware may alter domain-tool calls and state any required submission or trajectory-metadata disclosure.
Written by the indexing model from the issue text.
Description
We are planning a paired Agent Learning Track study using a custom BaseLLMClient + BaseAgent.
Both conditions would:
- build the learning artifact only from
datasets/train_task_trajectories/; - expose the same read-only
retrieve_learnings(query, top_k=3) -> list[str]; - present the same retrieved strings to the agent model;
- leave benchmark prompts, protocol files, task files, environments, domain tools, tool names, and argument keys unchanged; and
- let the STATE-Bench harness execute every domain tool.
The treatment condition adds deterministic middleware inside the custom BaseAgent, after the model proposes a domain-tool call but before the agent returns its AgentToolCallRequest. The middleware reads only:
- learning strings already returned by
retrieve_learnings, and - tool results already present in the current canonical conversation.
When a retrieved relation names a protected argument and a unique, type-compatible witness exists in the current run, the middleware may replace that argument value before returning the request. If the relation does not apply, the request passes through unchanged. It does not call the provided domain tool_handlers, inspect held-out task definitions or requirements, or mutate benchmark state. The benchmark still validates and executes the resulting request normally.
Issue #26 indicates that tools related to retrieving and using learnings may be added or modified. Would this deterministic pre-execution use of retrieved learnings be compliant with the Agent Learning Track and eligible for the official leaderboard, or must learnings affect domain-tool calls only through the model interpretation of the returned strings?
If compliant, is any additional method or trajectory-metadata disclosure required for submission?
- Dominant language
- Python
- Stars
- 91
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/STATE-Bench
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
microsoft/STATE-Bench#36 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 58/100
microsoft/STATE-Bench#52 ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
microsoft/STATE-Bench#51 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
microsoft/STATE-Bench#50 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
microsoft/STATE-Bench#48 · 2 comments ·
All issues in microsoft/STATE-Bench
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100