[BUG] declarative_flow AGENTS.md tells agents to `crewai run` before `crewai eval` and to log in, though eval runs the flow itself

オープン 初心者向け
#9,592 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
1/5
見積もり時間
1時間未満
初心者へのやさしさ
89/100
issue の種類
ドキュメント
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
documentation

調査の方向性

lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.mdの119~121行付近を読み、その指示をlib/cli/src/crewai_cli/templates/AGENTS.mdの指示およびeval_crew.pyに記載された動作と比較してください。declarative flowの指示を更新し、crewai evalがいつflowを実行するのか、tracingの設定が評価にどう影響するのか、briefの後に何をすべきかを説明してください。テンプレートが、最初にcrewai runを実行するようエージェントに指示したり、不必要なログインを求めたりしなくなれば完了です。

索引モデルが issue の本文から書いたものです。

説明

bug
Description
The `AGENTS.md` that `crewai create flow` copies into a declarative flow project (`lib/cli/src/crewai_cli/create_flow.py:232-251`) describes `crewai eval` the wrong way. At 86b9b09edd, eval already runs the flow itself (traced) when nothing is traced yet or the project changed. The template still tells the coding agent to run `crewai run` first and implies it has to log in. The generic template, `lib/cli/src/crewai_cli/templates/AGENTS.md:40`, already gets this right ("no `crewai run` in between").

There are two places in `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md`:

**1. Line 119: how eval picks the run, and when it is traced**

```
Run `crewai eval` after a change: it grades the project's last traced run — a goal gate, a 1–5 grade per area, and a link to the full report. It grades a run that already happened, so after changing code run the flow again first (`crewai run`, with `CREWAI_TRACING_ENABLED=true` in `.env`; a run with no terminal is only traced when logged in with `crewai login`), then `crewai eval`. In a terminal with no traced run recorded, it offers to run the flow once with tracing on.
```

What the code does:
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:942-952` (`no_traced_run` / `project_changed`) and `:393` (`_why_run_the_crew`): eval decides to run the flow whenever there is no traced run or the project changed, with or without a terminal. `:404-415` then runs it before grading.
- `eval_crew.py:1177-1191`: if tracing was never set and there is no terminal, eval turns tracing on by itself. At a terminal it asks first. It stops only when tracing is explicitly off (`CREWAI_TRACING_ENABLED=false`). In that case `:1124-1131` and `:270-274` (`TRACING_OFF_UNATTENDED`) tell the agent to set it to `true`.
- `eval_crew.py:207-212` (`NOT_TRACED`): a flow with `tracing=False` in code can't be evaluated.

Effect: an agent that isn't logged in reads that runs without a terminal are only traced after login. It either stops at `crewai login`, which eval doesn't need, or it runs an untraced `crewai run`, and eval then runs the flow again anyway.

**2. Line 121: what to do after the brief**

```
- Run by a coding agent or script (no terminal), it prints a **markdown brief** after the verdict: what failed and the change to make. Act on its "what failed" changes, then `crewai run` and `crewai eval` again. (An older AMP may print only a link to the brief, or nothing.)
```

What the code does: same as above. After the edits, `crewai eval` sees that the project changed and runs the flow itself (`eval_crew.py:393`, `:404-415`, `:942-952`).

Effect: every fix round runs the flow twice. The first run is `crewai run`, which is often untraced when there is no terminal and nobody is logged in. The second run happens inside `crewai eval`. That doubles the LLM cost and the time for each round.
Steps to Reproduce
1. Create a declarative flow project with `crewai create flow` so that `declarative_flow/AGENTS.md` is copied into the project root.
2. Have a coding agent with no terminal and no login follow the "Evaluate" instructions on lines 119-121.
3. The agent edits the flow, runs `crewai run` (untraced), then runs `crewai eval`.
4. `crewai eval` finds that the project changed since its last traced run and runs the flow a second time before grading it (`eval_crew.py:942-952`, `:404-415`). Or, following line 119, the agent stops to run `crewai login` first.
Expected behavior
The template should say what eval actually does. After a change, run only `crewai eval`, with no `crewai run` first and no login needed. It should also say when tracing gets in the way: `CREWAI_TRACING_ENABLED=false` in `.env`, or `tracing=False` in code. That would match `templates/AGENTS.md:40`.
Screenshots/Code snippets
Current text, `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:119`:

```
It grades a run that already happened, so after changing code run the flow again first (`crewai run`, with `CREWAI_TRACING_ENABLED=true` in `.env`; a run with no terminal is only traced when logged in with `crewai login`), then `crewai eval`.
```

Current text, `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:121`:

```
Act on its "what failed" changes, then `crewai run` and `crewai eval` again.
```
Operating System

Other (specify in additional context)

Python Version

3.12

crewAI Version

commit 86b9b09edd

crewAI Tools Version

Not involved (checked at crewAI commit 86b9b09edd)

Virtual Environment

Venv

Evidence
- `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:119` and `:121`: the text above
- `lib/cli/src/crewai_cli/create_flow.py:232-251`: copies `declarative_flow/AGENTS.md` into the project root
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:393`: `_why_run_the_crew`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:942-952`: `no_traced_run` / `project_changed`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:404-415`: runs the crew before grading
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:1177-1191`: tracing explicitly off → ask at a terminal or stop; never set → turn it on when there is no terminal, ask at a terminal
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:1124-1131`, `:270-274`: `TRACING_OFF_UNATTENDED`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:207-212`: `NOT_TRACED` (`tracing=False`)
- `lib/cli/src/crewai_cli/templates/AGENTS.md:40`: the generic template already says "no `crewai run` in between"
Possible Solution
Line 119:

```
Run `crewai eval` after a change: it grades the project's last traced run — a goal gate, a 1–5 grade per area, and a link to the full report. When there is no traced run, or the project changed since its last one, it runs the flow itself first, traced, and grades that run, so no `crewai run` is needed first, logged in or out. With no terminal it turns tracing on for the project if nobody has set it; if the project set `CREWAI_TRACING_ENABLED=false`, it stops and says to set it to `true` in `.env` (at a terminal it asks). A flow that sets `tracing=False` in code cannot be evaluated.
```

Line 121:

```
- Run by a coding agent or script (no terminal), it prints a **markdown brief** after the verdict: what failed and the change to make. Act on its "what failed" changes, then `crewai eval` again; no `crewai run` in between (it runs the flow itself when the project changed). (An older AMP may print only a link to the brief, or nothing.)
```
Additional context

This is a documentation and template problem that depends only on the code, not on the OS, Python version or virtual environment. The OS, Python and environment fields are filled in only because the form requires them. Everything above was checked against commit 86b9b09edd. Only the declarative flow template is affected. The generic templates/AGENTS.md already gives the right instruction.

主要言語
Python
スター
60.9k
フォーク
9.2k
平均マージ
1日 9時間
マージ済み PR(30日)
226

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

crewAIInc/crewAI のほかの issue

crewAIInc/crewAI の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。