fix(backtests): an LLM "sell 10" sells the whole position

Closed Beginner friendly
  • Is something wrong?
#661 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

A pull request for this has already been merged.

  • #667 by @subidhkhanal — merged

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend, testing

Research direction

Start with the sell branch in domain/backtesting/portfolio_manager.py at lines 981–995 and compare its sizing with actions_to_executable in infrastructure/llm/validator.py at line 391. Check how pipeline_runner.py passes size_pct as position_size, and inspect the T+1 deferral record. Done means a request to sell 10 of 500 shares sells 10, missing or zero size still sells the whole position, and requests above the sellable quantity are capped while preserving the requested amount for deferral.

Written by the indexing model from the issue text.

Description

The bug. In an LLM backtest, a SELL always closes the whole position. The model is told to say how many shares to sell, but the code never reads that number. So "sell 10 AAPL" while holding 500 sells all 500.

The model is told one thing and the engine does another:

  • The prompt (infrastructure/llm/validator.py:635, in the default SAFE_TRADING_PROMPT) asks for a share count: 
    - For SELL, position_size should be the shares to sell if available; otherwise use 1.
  • The sell branch (domain/backtesting/portfolio_manager.py:981-995) ignores position_size and sells everything sellable: 
    elif action_type == "sell":
    held = self.positions.get(symbol, 0)
    sellable = self.sellable_shares(symbol)
    ...
    action = {"symbol": symbol, "action": "sell", "shares": sellable, ...}

The other decision path already gets this right. actions_to_executable (validator.py:391), used by external agents, sells what was asked for and falls back to the whole position only when no size is given:

sell_shares = min(decision.position_size or positions[symbol], positions[symbol])

Pipeline runs are affected too. pipeline_runner.py:283-296 turns a risk_actions entry's size_pct into a SELL position_size. For example, {"action": "trail", "size_pct": 0.25} is meant as a partial trim. Because the sell branch ignores the size, it becomes a full exit.

Impact. Every partial trim the model asks for becomes a full liquidation. LLM equity curves, and the leaderboard entries built from them, are more all-in/all-out than the model's own decisions, so they don't show the strategy the model actually chose.

Suggested fix. Size the sell the way actions_to_executable does, capped at the sellable quantity:

requested = <parsed position_size> or held
shares = min(requested, sellable)

Keep the T+1 deferral record: when requested > sellable, record the requested amount as requested_shares.

Test. Hold 500 AAPL, send {"action": "sell", "symbol": "AAPL", "position_size": 10}, and assert one sell of 10 shares. Also check that a missing or zero position_size still sells the whole position.

Dominant language
Python
Stars
767
Forks
156
Avg merge
1d 18h
Merged PRs (30d)
73

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Open-Finance-Lab/AgenticTrading

All issues in Open-Finance-Lab/AgenticTrading

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.