Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Why Do Results from MCP-Trained Models Differ Greatly Between generate_benchmarks.py and train.py

Open
  • Is something wrong?
#417 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
32/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
machine-learning, testing-qa

Research direction

Start by comparing generate_benchmarks.py and train.py, focusing on how each handles the complete_task tool and determines task success. Trace the evaluation path for the MCP-trained model and reproduce the discrepancy with the supplied vLLM and LoRA configuration. Done means confirming whether the completion criteria differ and documenting or correcting the cause of the validation gap.

Written by the indexing model from the issue text.

Description

question

Why does a model trained via MCP show a significant discrepancy in results when tested using generate_benchmarks.py, compared to the outcomes from train.py?
A preliminary investigation indicates that the root cause may be related to the following logic: In generate_benchmarks.py, the model must call the complete_task function to be deemed as having finished a task. However, there is no such logic implemented in train.py. Is this the reason for the large result deviation?

generate_benchmarks.py

    qwen3_4b_instruct = art.Model(
        name="qwen3-4b-instruct",
        project=server,
        inference_model_name="qwen3-4b-instruct",
        inference_base_url="http://localhost:8082/v1", #http://localhost:8082/v1
        inference_api_key="dummy",  # API key
        inference_timeout=3600,
    )

source /*****/miniconda3/bin/activate ART
BASE_MODEL_PATH="/*****/Qwen3-4B-Instruct-2507"
LORA_PATH="/*****/examples/mcp-rl/.art/mcp-agent-training/models/mcp-4b-001/checkpoints/0017"


CUDA_VISIBLE_DEVICES=2,3 python -m vllm.entrypoints.openai.api_server \
    --model "$BASE_MODEL_PATH" \
    --served-model-name "mcp-4b-001-finetuned" \
    --enable-lora \
    --lora-modules mcp-4b-001="$LORA_PATH" \
    --host 0.0.0.0 \
    --port 8082 \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --max-model-len 16384 \
    --tensor-parallel-size 2

In the generate_benchmarks.py

    mcp-4b-001-finetuned = art.Model(
        name="mcp-4b-001-finetuned",
        project=server,
        inference_model_name="mcp-4b-001-finetuned",
        inference_base_url="http://localhost:8082/v1", #http://localhost:8082/v1
        inference_api_key="dummy",  
        inference_timeout=3600)

While this method has a certain degree of effectiveness, there is still a significant gap between its current performance and the validation results obtained during training.

I would like to know: during the training process, is it also mandatory for the model to output the "complete task" tool to be considered a successful completion of the task? Because when I used your benchmark, the trained large model tended not to call the "complete task" tool to end the task, resulting in an evaluation success rate of 0.

Dominant language
Python
Stars
10.8k
Forks
989
Avg merge
11h 38m
Merged PRs (30d)
104

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenPipe/ART

All issues in OpenPipe/ART

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.