[BUG] declarative_flow AGENTS.md tells agents to `crewai run` before `crewai eval` and to log in, though eval runs the flow itself

Đang mở Phù hợp với người mới
#9,592 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
1/5
Thời gian dự kiến
Dưới một giờ
Mức phù hợp với người mới
89/100
Loại issue
Tài liệu
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
documentation

Hướng nghiên cứu

Đọc lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md quanh dòng 119–121 và so sánh hướng dẫn trong đó với lib/cli/src/crewai_cli/templates/AGENTS.md cũng như hành vi được nêu trong eval_crew.py. Cập nhật hướng dẫn cho declarative flow để giải thích khi nào crewai eval chạy flow, các thiết lập tracing ảnh hưởng đến việc đánh giá như thế nào và cần làm gì sau một brief. Hoàn tất khi template không còn yêu cầu agent chạy crewai run trước hoặc đăng nhập không cần thiết.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug
Description
The `AGENTS.md` that `crewai create flow` copies into a declarative flow project (`lib/cli/src/crewai_cli/create_flow.py:232-251`) describes `crewai eval` the wrong way. At 86b9b09edd, eval already runs the flow itself (traced) when nothing is traced yet or the project changed. The template still tells the coding agent to run `crewai run` first and implies it has to log in. The generic template, `lib/cli/src/crewai_cli/templates/AGENTS.md:40`, already gets this right ("no `crewai run` in between").

There are two places in `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md`:

**1. Line 119: how eval picks the run, and when it is traced**

```
Run `crewai eval` after a change: it grades the project's last traced run — a goal gate, a 1–5 grade per area, and a link to the full report. It grades a run that already happened, so after changing code run the flow again first (`crewai run`, with `CREWAI_TRACING_ENABLED=true` in `.env`; a run with no terminal is only traced when logged in with `crewai login`), then `crewai eval`. In a terminal with no traced run recorded, it offers to run the flow once with tracing on.
```

What the code does:
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:942-952` (`no_traced_run` / `project_changed`) and `:393` (`_why_run_the_crew`): eval decides to run the flow whenever there is no traced run or the project changed, with or without a terminal. `:404-415` then runs it before grading.
- `eval_crew.py:1177-1191`: if tracing was never set and there is no terminal, eval turns tracing on by itself. At a terminal it asks first. It stops only when tracing is explicitly off (`CREWAI_TRACING_ENABLED=false`). In that case `:1124-1131` and `:270-274` (`TRACING_OFF_UNATTENDED`) tell the agent to set it to `true`.
- `eval_crew.py:207-212` (`NOT_TRACED`): a flow with `tracing=False` in code can't be evaluated.

Effect: an agent that isn't logged in reads that runs without a terminal are only traced after login. It either stops at `crewai login`, which eval doesn't need, or it runs an untraced `crewai run`, and eval then runs the flow again anyway.

**2. Line 121: what to do after the brief**

```
- Run by a coding agent or script (no terminal), it prints a **markdown brief** after the verdict: what failed and the change to make. Act on its "what failed" changes, then `crewai run` and `crewai eval` again. (An older AMP may print only a link to the brief, or nothing.)
```

What the code does: same as above. After the edits, `crewai eval` sees that the project changed and runs the flow itself (`eval_crew.py:393`, `:404-415`, `:942-952`).

Effect: every fix round runs the flow twice. The first run is `crewai run`, which is often untraced when there is no terminal and nobody is logged in. The second run happens inside `crewai eval`. That doubles the LLM cost and the time for each round.
Steps to Reproduce
1. Create a declarative flow project with `crewai create flow` so that `declarative_flow/AGENTS.md` is copied into the project root.
2. Have a coding agent with no terminal and no login follow the "Evaluate" instructions on lines 119-121.
3. The agent edits the flow, runs `crewai run` (untraced), then runs `crewai eval`.
4. `crewai eval` finds that the project changed since its last traced run and runs the flow a second time before grading it (`eval_crew.py:942-952`, `:404-415`). Or, following line 119, the agent stops to run `crewai login` first.
Expected behavior
The template should say what eval actually does. After a change, run only `crewai eval`, with no `crewai run` first and no login needed. It should also say when tracing gets in the way: `CREWAI_TRACING_ENABLED=false` in `.env`, or `tracing=False` in code. That would match `templates/AGENTS.md:40`.
Screenshots/Code snippets
Current text, `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:119`:

```
It grades a run that already happened, so after changing code run the flow again first (`crewai run`, with `CREWAI_TRACING_ENABLED=true` in `.env`; a run with no terminal is only traced when logged in with `crewai login`), then `crewai eval`.
```

Current text, `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:121`:

```
Act on its "what failed" changes, then `crewai run` and `crewai eval` again.
```
Operating System

Other (specify in additional context)

Python Version

3.12

crewAI Version

commit 86b9b09edd

crewAI Tools Version

Not involved (checked at crewAI commit 86b9b09edd)

Virtual Environment

Venv

Evidence
- `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:119` and `:121`: the text above
- `lib/cli/src/crewai_cli/create_flow.py:232-251`: copies `declarative_flow/AGENTS.md` into the project root
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:393`: `_why_run_the_crew`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:942-952`: `no_traced_run` / `project_changed`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:404-415`: runs the crew before grading
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:1177-1191`: tracing explicitly off → ask at a terminal or stop; never set → turn it on when there is no terminal, ask at a terminal
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:1124-1131`, `:270-274`: `TRACING_OFF_UNATTENDED`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:207-212`: `NOT_TRACED` (`tracing=False`)
- `lib/cli/src/crewai_cli/templates/AGENTS.md:40`: the generic template already says "no `crewai run` in between"
Possible Solution
Line 119:

```
Run `crewai eval` after a change: it grades the project's last traced run — a goal gate, a 1–5 grade per area, and a link to the full report. When there is no traced run, or the project changed since its last one, it runs the flow itself first, traced, and grades that run, so no `crewai run` is needed first, logged in or out. With no terminal it turns tracing on for the project if nobody has set it; if the project set `CREWAI_TRACING_ENABLED=false`, it stops and says to set it to `true` in `.env` (at a terminal it asks). A flow that sets `tracing=False` in code cannot be evaluated.
```

Line 121:

```
- Run by a coding agent or script (no terminal), it prints a **markdown brief** after the verdict: what failed and the change to make. Act on its "what failed" changes, then `crewai eval` again; no `crewai run` in between (it runs the flow itself when the project changed). (An older AMP may print only a link to the brief, or nothing.)
```
Additional context

This is a documentation and template problem that depends only on the code, not on the OS, Python version or virtual environment. The OS, Python and environment fields are filled in only because the form requires them. Everything above was checked against commit 86b9b09edd. Only the declarative flow template is affected. The generic templates/AGENTS.md already gives the right instruction.

Ngôn ngữ chính
Python
Star
60.9k
Fork
9.2k
Merge trung bình
1 ngày 9 giờ
Pull request đã merge (30 ngày)
226

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của crewAIInc/crewAI

Tất cả issue của crewAIInc/crewAI

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.