refactor(llm): check the AI's trading answer with one shared schema — today two code paths read it differently

Open
  • Is something wrong?
#665 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend, testing

Research direction

Start by reading infrastructure/llm/validator.py and domain/backtesting/portfolio_manager.py, then trace pipeline_output_to_decision in infrastructure/llm/pipeline_runner.py and the single-prompt path. The plan splits the work into four PRs, with sell sizing first; the full issue also calls for shared parsing, reliable usage reporting, and shared tests using scripted AI answers. Done means both backtest paths use the checked schema and the listed malformed answers are covered offline.

Written by the indexing model from the issue text.

Description

The problem

When the AI answers "what should I trade?", our code reads that answer in two different ways, and they disagree.

Way 1: the backtest trade loop. This is used by every AI backtest started from the website (the pipeline path) and by leaderboard runs (the single-prompt path).

  • The single-prompt path reads only the actions label.
  • The pipeline path also accepts orders and risk_actions, and turns them into actions (pipeline_output_to_decision in infrastructure/llm/pipeline_runner.py).
  • Both then go through the same trade loop: make_trading_decision_with_llm in domain/backtesting/portfolio_manager.py.
  • That loop barely checks the answer. It assumes every field is correctly typed.

Way 2: external agents. These are /api/v1, /api/v2 and the AI Hedge Fund connector.

  • Their answers are checked by a strict schema: LLMTradingDecision and parse_actions_payload in infrastructure/llm/validator.py.
  • That schema already handles the bad cases correctly.

So the same answer can be traded one way by the backtest loop and another way by external agents. Each time we fix one bug in the loop, a similar one appears next to it.

What goes wrong today

Every row below was tested on main by running the real code with a scripted AI answer.

AI answer Backtest trade loop External-agent schema
"sell 10" while holding 500 sells all 500, on both the pipeline and single-prompt paths (#661) sells 10 ✅
action: "buy " (extra space) places nothing, but still counts it as an AI decision (#658) rejected with a clear error
confidence: "0.8" (number as text) crashes; the AI's trade is thrown away and a paid run stops accepted ✅
confidence: "high" or null crashes; trade thrown away, paid run stops (also #663 on the orders label) rejected with a clear error
reasoning: null crashes; trade thrown away, paid run stops rejected with a clear error
trades under an unknown label, e.g. {"decision": {"actions": [...]}} single-prompt strict mode: counted as "AI chose to hold", trades lost. Pipeline: paid run stops. rejected with a clear error
Why it matters
  • Users pay for runs that crash on small formatting slips. Every AI call before the crash is already charged.
  • Trades the AI never asked for. "Sell 10" becomes "sell everything" in every AI backtest, so results don't reflect what the AI decided.
  • Leaderboard curves quietly mix in rule-based trades whenever an answer is thrown away.
  • The AI-decision counter (llm_decisions) can be wrong. A lost answer can be counted as a real AI decision.
Proposal

Make the backtest trade loop use one shared, checked schema, like external agents already do:

  1. One schema for a trade (symbol, action, position_size, confidence, reasoning, optional stop and take-profit prices), based on LLMTradingDecision. It should be forgiving about small slips:
    • trim and lowercase action;
    • accept numbers written as text;
    • use the normal default when confidence is missing or not a number;
    • treat a missing reasoning as empty text.
  2. One reader for the answer's labels. Accept actions, orders and risk_actions, and return one of three clear results:
    • a valid decision;
    • "the AI chose to hold", which only an explicit actions: [] counts as;
    • an unusable answer.
  3. One sizing rule. A buy uses position_size. A sell uses min(position_size or shares held, shares sellable).
  4. Never lose usage. The pipeline always reports the step's AI calls and tokens, even when the answer can't be used.
  5. One shared test table of messy real-world answers, run against the schema, plus a test that each path really uses it. The table covers the cases above, plus code fences, extra text around the JSON, duplicates, NaN/Infinity and non-object entries.

Not included: prompt wording, provider adapters, and the Trading Algo script, which can't be reached from the website.

Testing: everything can be tested offline with scripted AI answers. No API key is needed.

Related: #658, #661, #663.

Plan

I will do this in 4 separate PRs, each small enough to review on its own:

  1. Sell sizing (#661): a small PR, and the most important fix.
  2. Safe number conversion, plus always recording usage: fixes #663 and the confidence crash.
  3. One label reader: only an explicit actions: [] counts as "hold".
  4. Move the trade loop onto the shared schema, with the shared test table.
Dominant language
Python
Stars
767
Forks
156
Avg merge
1d 18h
Merged PRs (30d)
73

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Open-Finance-Lab/AgenticTrading

All issues in Open-Finance-Lab/AgenticTrading

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.