[BUG] declarative_flow AGENTS.md tells agents to `crewai run` before `crewai eval` and to log in, though eval runs the flow itself
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 1/5
- 見積もり時間
- 1時間未満
- 初心者へのやさしさ
- 89/100
- issue の種類
- ドキュメント
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- python
調査の方向性
lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.mdの119~121行付近を読み、その指示をlib/cli/src/crewai_cli/templates/AGENTS.mdの指示およびeval_crew.pyに記載された動作と比較してください。declarative flowの指示を更新し、crewai evalがいつflowを実行するのか、tracingの設定が評価にどう影響するのか、briefの後に何をすべきかを説明してください。テンプレートが、最初にcrewai runを実行するようエージェントに指示したり、不必要なログインを求めたりしなくなれば完了です。
索引モデルが issue の本文から書いたものです。
説明
Description
The `AGENTS.md` that `crewai create flow` copies into a declarative flow project (`lib/cli/src/crewai_cli/create_flow.py:232-251`) describes `crewai eval` the wrong way. At 86b9b09edd, eval already runs the flow itself (traced) when nothing is traced yet or the project changed. The template still tells the coding agent to run `crewai run` first and implies it has to log in. The generic template, `lib/cli/src/crewai_cli/templates/AGENTS.md:40`, already gets this right ("no `crewai run` in between").
There are two places in `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md`:
**1. Line 119: how eval picks the run, and when it is traced**
```
Run `crewai eval` after a change: it grades the project's last traced run — a goal gate, a 1–5 grade per area, and a link to the full report. It grades a run that already happened, so after changing code run the flow again first (`crewai run`, with `CREWAI_TRACING_ENABLED=true` in `.env`; a run with no terminal is only traced when logged in with `crewai login`), then `crewai eval`. In a terminal with no traced run recorded, it offers to run the flow once with tracing on.
```
What the code does:
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:942-952` (`no_traced_run` / `project_changed`) and `:393` (`_why_run_the_crew`): eval decides to run the flow whenever there is no traced run or the project changed, with or without a terminal. `:404-415` then runs it before grading.
- `eval_crew.py:1177-1191`: if tracing was never set and there is no terminal, eval turns tracing on by itself. At a terminal it asks first. It stops only when tracing is explicitly off (`CREWAI_TRACING_ENABLED=false`). In that case `:1124-1131` and `:270-274` (`TRACING_OFF_UNATTENDED`) tell the agent to set it to `true`.
- `eval_crew.py:207-212` (`NOT_TRACED`): a flow with `tracing=False` in code can't be evaluated.
Effect: an agent that isn't logged in reads that runs without a terminal are only traced after login. It either stops at `crewai login`, which eval doesn't need, or it runs an untraced `crewai run`, and eval then runs the flow again anyway.
**2. Line 121: what to do after the brief**
```
- Run by a coding agent or script (no terminal), it prints a **markdown brief** after the verdict: what failed and the change to make. Act on its "what failed" changes, then `crewai run` and `crewai eval` again. (An older AMP may print only a link to the brief, or nothing.)
```
What the code does: same as above. After the edits, `crewai eval` sees that the project changed and runs the flow itself (`eval_crew.py:393`, `:404-415`, `:942-952`).
Effect: every fix round runs the flow twice. The first run is `crewai run`, which is often untraced when there is no terminal and nobody is logged in. The second run happens inside `crewai eval`. That doubles the LLM cost and the time for each round.
Steps to Reproduce
1. Create a declarative flow project with `crewai create flow` so that `declarative_flow/AGENTS.md` is copied into the project root.
2. Have a coding agent with no terminal and no login follow the "Evaluate" instructions on lines 119-121.
3. The agent edits the flow, runs `crewai run` (untraced), then runs `crewai eval`.
4. `crewai eval` finds that the project changed since its last traced run and runs the flow a second time before grading it (`eval_crew.py:942-952`, `:404-415`). Or, following line 119, the agent stops to run `crewai login` first.
Expected behavior
The template should say what eval actually does. After a change, run only `crewai eval`, with no `crewai run` first and no login needed. It should also say when tracing gets in the way: `CREWAI_TRACING_ENABLED=false` in `.env`, or `tracing=False` in code. That would match `templates/AGENTS.md:40`.
Screenshots/Code snippets
Current text, `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:119`:
```
It grades a run that already happened, so after changing code run the flow again first (`crewai run`, with `CREWAI_TRACING_ENABLED=true` in `.env`; a run with no terminal is only traced when logged in with `crewai login`), then `crewai eval`.
```
Current text, `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:121`:
```
Act on its "what failed" changes, then `crewai run` and `crewai eval` again.
```
Operating System
Other (specify in additional context)
Python Version
3.12
crewAI Version
commit 86b9b09edd
crewAI Tools Version
Not involved (checked at crewAI commit 86b9b09edd)
Virtual Environment
Venv
Evidence
- `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:119` and `:121`: the text above
- `lib/cli/src/crewai_cli/create_flow.py:232-251`: copies `declarative_flow/AGENTS.md` into the project root
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:393`: `_why_run_the_crew`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:942-952`: `no_traced_run` / `project_changed`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:404-415`: runs the crew before grading
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:1177-1191`: tracing explicitly off → ask at a terminal or stop; never set → turn it on when there is no terminal, ask at a terminal
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:1124-1131`, `:270-274`: `TRACING_OFF_UNATTENDED`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:207-212`: `NOT_TRACED` (`tracing=False`)
- `lib/cli/src/crewai_cli/templates/AGENTS.md:40`: the generic template already says "no `crewai run` in between"
Possible Solution
Line 119:
```
Run `crewai eval` after a change: it grades the project's last traced run — a goal gate, a 1–5 grade per area, and a link to the full report. When there is no traced run, or the project changed since its last one, it runs the flow itself first, traced, and grades that run, so no `crewai run` is needed first, logged in or out. With no terminal it turns tracing on for the project if nobody has set it; if the project set `CREWAI_TRACING_ENABLED=false`, it stops and says to set it to `true` in `.env` (at a terminal it asks). A flow that sets `tracing=False` in code cannot be evaluated.
```
Line 121:
```
- Run by a coding agent or script (no terminal), it prints a **markdown brief** after the verdict: what failed and the change to make. Act on its "what failed" changes, then `crewai eval` again; no `crewai run` in between (it runs the flow itself when the project changed). (An older AMP may print only a link to the brief, or nothing.)
```
Additional context
This is a documentation and template problem that depends only on the code, not on the OS, Python version or virtual environment. The OS, Python and environment fields are filled in only because the form requires them. Everything above was checked against commit 86b9b09edd. Only the declarative flow template is affected. The generic templates/AGENTS.md already gives the right instruction.
- 主要言語
- Python
- スター
- 60.9k
- フォーク
- 9.2k
- 平均マージ
- 1日 9時間
- マージ済み PR(30日)
- 226
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
crewAIInc/crewAI のほかの issue
-
[BUG] Agent.copy() shares the LLM's additional_params and token usage対応中かも @HanYuanLu が今日担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
crewAIInc/crewAI#9587 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
[BUG] CrewJSONEncoder saves enums as "Status.DONE" instead of their value対応中かも @nurlanmnn が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
[FEATURE] Document a single-agent workflow with skills対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 67/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
メンテナーはふだん 1 日以内に返信
-
feature-request
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
crewAIInc/crewAI#9200 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
crewAIInc/crewAI の issue をすべて見る
似ている issue
-
enhancement
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
DoRA magnitude vectors are not loaded by Transformers load_adapter / from_pretrained(adapter_dir)オープンpost-v1
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
huggingface/peft#4208 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
approved correction metadata
難易度 1/5 1時間未満 初心者へのやさしさ 76/100
acl-org/acl-anthology#10996 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
needs-triage
難易度 1/5 1〜3時間 初心者へのやさしさ 85/100
gke-labs/kube-agents#6620 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
ashhart/TensorFold#4150 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信