[BUG] declarative_flow AGENTS.md tells agents to `crewai run` before `crewai eval` and to log in, though eval runs the flow itself

Abierto Apto para principiantes
#9,592 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
1/5
Tiempo estimado
Menos de una hora
Aptitud para principiantes
89/100
Tipo de issue
Documentación
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python

Línea de trabajo

Lee lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md alrededor de las líneas 119–121 y compara sus indicaciones con las de lib/cli/src/crewai_cli/templates/AGENTS.md y el comportamiento descrito en eval_crew.py. Actualiza las instrucciones de declarative flow para explicar cuándo crewai eval ejecuta el flow, cómo afectan los ajustes de tracing a la evaluación y qué hacer después de un brief. El trabajo estará terminado cuando la plantilla ya no indique a los agentes que ejecuten primero crewai run ni que inicien sesión sin necesidad.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
Description
The `AGENTS.md` that `crewai create flow` copies into a declarative flow project (`lib/cli/src/crewai_cli/create_flow.py:232-251`) describes `crewai eval` the wrong way. At 86b9b09edd, eval already runs the flow itself (traced) when nothing is traced yet or the project changed. The template still tells the coding agent to run `crewai run` first and implies it has to log in. The generic template, `lib/cli/src/crewai_cli/templates/AGENTS.md:40`, already gets this right ("no `crewai run` in between").

There are two places in `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md`:

**1. Line 119: how eval picks the run, and when it is traced**

```
Run `crewai eval` after a change: it grades the project's last traced run — a goal gate, a 1–5 grade per area, and a link to the full report. It grades a run that already happened, so after changing code run the flow again first (`crewai run`, with `CREWAI_TRACING_ENABLED=true` in `.env`; a run with no terminal is only traced when logged in with `crewai login`), then `crewai eval`. In a terminal with no traced run recorded, it offers to run the flow once with tracing on.
```

What the code does:
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:942-952` (`no_traced_run` / `project_changed`) and `:393` (`_why_run_the_crew`): eval decides to run the flow whenever there is no traced run or the project changed, with or without a terminal. `:404-415` then runs it before grading.
- `eval_crew.py:1177-1191`: if tracing was never set and there is no terminal, eval turns tracing on by itself. At a terminal it asks first. It stops only when tracing is explicitly off (`CREWAI_TRACING_ENABLED=false`). In that case `:1124-1131` and `:270-274` (`TRACING_OFF_UNATTENDED`) tell the agent to set it to `true`.
- `eval_crew.py:207-212` (`NOT_TRACED`): a flow with `tracing=False` in code can't be evaluated.

Effect: an agent that isn't logged in reads that runs without a terminal are only traced after login. It either stops at `crewai login`, which eval doesn't need, or it runs an untraced `crewai run`, and eval then runs the flow again anyway.

**2. Line 121: what to do after the brief**

```
- Run by a coding agent or script (no terminal), it prints a **markdown brief** after the verdict: what failed and the change to make. Act on its "what failed" changes, then `crewai run` and `crewai eval` again. (An older AMP may print only a link to the brief, or nothing.)
```

What the code does: same as above. After the edits, `crewai eval` sees that the project changed and runs the flow itself (`eval_crew.py:393`, `:404-415`, `:942-952`).

Effect: every fix round runs the flow twice. The first run is `crewai run`, which is often untraced when there is no terminal and nobody is logged in. The second run happens inside `crewai eval`. That doubles the LLM cost and the time for each round.
Steps to Reproduce
1. Create a declarative flow project with `crewai create flow` so that `declarative_flow/AGENTS.md` is copied into the project root.
2. Have a coding agent with no terminal and no login follow the "Evaluate" instructions on lines 119-121.
3. The agent edits the flow, runs `crewai run` (untraced), then runs `crewai eval`.
4. `crewai eval` finds that the project changed since its last traced run and runs the flow a second time before grading it (`eval_crew.py:942-952`, `:404-415`). Or, following line 119, the agent stops to run `crewai login` first.
Expected behavior
The template should say what eval actually does. After a change, run only `crewai eval`, with no `crewai run` first and no login needed. It should also say when tracing gets in the way: `CREWAI_TRACING_ENABLED=false` in `.env`, or `tracing=False` in code. That would match `templates/AGENTS.md:40`.
Screenshots/Code snippets
Current text, `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:119`:

```
It grades a run that already happened, so after changing code run the flow again first (`crewai run`, with `CREWAI_TRACING_ENABLED=true` in `.env`; a run with no terminal is only traced when logged in with `crewai login`), then `crewai eval`.
```

Current text, `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:121`:

```
Act on its "what failed" changes, then `crewai run` and `crewai eval` again.
```
Operating System

Other (specify in additional context)

Python Version

3.12

crewAI Version

commit 86b9b09edd

crewAI Tools Version

Not involved (checked at crewAI commit 86b9b09edd)

Virtual Environment

Venv

Evidence
- `lib/cli/src/crewai_cli/templates/declarative_flow/AGENTS.md:119` and `:121`: the text above
- `lib/cli/src/crewai_cli/create_flow.py:232-251`: copies `declarative_flow/AGENTS.md` into the project root
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:393`: `_why_run_the_crew`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:942-952`: `no_traced_run` / `project_changed`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:404-415`: runs the crew before grading
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:1177-1191`: tracing explicitly off → ask at a terminal or stop; never set → turn it on when there is no terminal, ask at a terminal
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:1124-1131`, `:270-274`: `TRACING_OFF_UNATTENDED`
- `lib/cli/src/crewai_cli/experimental/eval_crew.py:207-212`: `NOT_TRACED` (`tracing=False`)
- `lib/cli/src/crewai_cli/templates/AGENTS.md:40`: the generic template already says "no `crewai run` in between"
Possible Solution
Line 119:

```
Run `crewai eval` after a change: it grades the project's last traced run — a goal gate, a 1–5 grade per area, and a link to the full report. When there is no traced run, or the project changed since its last one, it runs the flow itself first, traced, and grades that run, so no `crewai run` is needed first, logged in or out. With no terminal it turns tracing on for the project if nobody has set it; if the project set `CREWAI_TRACING_ENABLED=false`, it stops and says to set it to `true` in `.env` (at a terminal it asks). A flow that sets `tracing=False` in code cannot be evaluated.
```

Line 121:

```
- Run by a coding agent or script (no terminal), it prints a **markdown brief** after the verdict: what failed and the change to make. Act on its "what failed" changes, then `crewai eval` again; no `crewai run` in between (it runs the flow itself when the project changed). (An older AMP may print only a link to the brief, or nothing.)
```
Additional context

This is a documentation and template problem that depends only on the code, not on the OS, Python version or virtual environment. The OS, Python and environment fields are filled in only because the form requires them. Everything above was checked against commit 86b9b09edd. Only the declarative flow template is affected. The generic templates/AGENTS.md already gives the right instruction.

Lenguaje dominante
Python
Estrellas
60.9k
Forks
9.2k
Merge medio
1 d 9 h
PR fusionados (30 d)
226

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de crewAIInc/crewAI

Todos los issues de crewAIInc/crewAI

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.