[Bug]: Unsupported expression syntax evaluates to None silently instead of raising

Abierto
  • ¿Algo no está bien?
#5,073 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

  • #5206 de @aarjang — cerrado sin fusionar

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
52/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, testing

Línea de trabajo

Start in src/specify_cli/workflows/expressions.py, especially the parser and evaluation points cited in the report: lines 304, 331, 715, 800, 884, 893, 930-976, and _safe_compare. Read the evaluator tests in tests/test_workflows.py and run the reproductions from the issue for exponent numbers, escaped quotes, chained indexing, and adjacent word operators. Done means each documented case evaluates as expected or raises a clear ValueError for unsupported syntax.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

triage-must-have
Bug Description

When the workflow expression evaluator meets syntax it does not parse, it does not raise. It treats the text as a variable path, which resolves to None (_resolve_dot_path, reached from src/specify_cli/workflows/expressions.py:969). A condition that uses such syntax then quietly gives the wrong result. The following cases were confirmed:

  1. Exponent-form numbers. Numeric literals and numeric strings are parsed with float() only when they contain a ., and otherwise with int() (expressions.py:930-935 for literals, expressions.py:972-976 for _coerce_number). So 1e3 is not read as a number: as a literal it is looked up as a variable and becomes None, and the string "1e3" stays a string in comparisons. Workflow input coercion does accept "1e3" as the number 1000 (src/specify_cli/workflows/engine.py:1449-1456), so the same text means different things in different places.
  2. Escaped quotes in string literals. A string literal is recognised only if its first matching quote is its last character (expressions.py:800), so 'it\'s' is looked up as a variable and becomes None. As a result, inputs.name == 'it\'s' is false for name = "it's".
  3. Chained indexing. The indexed-segment pattern accepts one index only (expressions.py:304), so matrix[0][1] is looked up as the literal key "matrix[0][1]" (expressions.py:331) and returns None.
  4. Word operators next to a quote. and, or and in are only found with a space on both sides (expressions.py:884, expressions.py:893, expressions.py:715). inputs.a=='x'and inputs.b is therefore split only at ==, its right side 'x'and inputs.b is looked up as a variable and becomes None, and the result is false.
Steps to Reproduce
  1. From a fresh clone of spec-kit, run uv sync.
  2. Save the script below as repro.py in the repository root.
  3. Run uv run python -I repro.py.
from specify_cli.workflows.base import StepContext
from specify_cli.workflows.engine import WorkflowEngine
from specify_cli.workflows.expressions import evaluate_condition, evaluate_expression

ctx = StepContext(
    inputs={"n": 5, "s": "1e3", "name": "it's", "a": "x", "b": True},
    steps={"plan": {"output": {"matrix": [["a", "b"], ["c", "d"]]}}},
)
cases = [
    ("1", "{{ 1e3 }}", "value"),
    ("1", "{{ inputs.n < 1e3 }}", "cond"),
    ("1", "{{ inputs.s > 100 }}", "cond"),
    ("2", r"{{ 'it\'s' }}", "value"),
    ("2", r"{{ inputs.name == 'it\'s' }}", "cond"),
    ("3", "{{ steps.plan.output.matrix[0] }}", "value"),
    ("3", "{{ steps.plan.output.matrix[0][1] }}", "value"),
    ("4", "{{ inputs.a == 'x' and inputs.b }}", "cond"),
    ("4", "{{ inputs.a=='x'and inputs.b }}", "cond"),
    ("4", "{{ 'x'in inputs.a }}", "cond"),
]
for num, expr, kind in cases:
    fn = evaluate_expression if kind == "value" else evaluate_condition
    print(f"[{num}] {expr} -> {fn(expr, ctx)!r}")
print("input coercion of '1e3' (type: number):",
      WorkflowEngine._coerce_input("x", "1e3", {"type": "number"}))
Expected Behavior

The module docstring describes the evaluator as "a safe Jinja2 subset" (src/specify_cli/workflows/expressions.py:3), and _evaluate_simple_expression lists "String and numeric literals" as supported (expressions.py:791). For each case, the expression should either evaluate as it would in Jinja2 or raise a ValueError naming the unsupported syntax. Malformed filters already get such a ValueError (docs/reference/workflows.md, "Types are validated, not coerced"). Specifically:

  1. {{ 1e3 }} should be 1000.0, and {{ inputs.n < 1e3 }} should be true for n = 5. {{ inputs.s > 100 }} with s = "1e3" should be true, in line with the _safe_compare docstring (expressions.py:1003-1012) and the input-coercion test tests/test_workflows.py:7092 (("1e3", 1000)).
  2. {{ 'it\'s' }} should be "it's", and the comparison should be true.
  3. {{ steps.plan.output.matrix[0][1] }} should be 'b'. The _resolve_dot_path docstring promises list indexing (expressions.py:312).
  4. {{ inputs.a=='x'and inputs.b }} should be true, like the spaced form. {{ 'x'in inputs.a }} should be true.
Actual Behavior
[1] {{ 1e3 }} -> None
[1] {{ inputs.n < 1e3 }} -> False
[1] {{ inputs.s > 100 }} -> False
[2] {{ 'it\'s' }} -> None
[2] {{ inputs.name == 'it\'s' }} -> False
[3] {{ steps.plan.output.matrix[0] }} -> ['a', 'b']
[3] {{ steps.plan.output.matrix[0][1] }} -> None
[4] {{ inputs.a == 'x' and inputs.b }} -> True
[4] {{ inputs.a=='x'and inputs.b }} -> False
[4] {{ 'x'in inputs.a }} -> False
input coercion of '1e3' (type: number): 1000
Specify CLI Version

1.2.4.dev0 (main at 27d14e2)

AI Agent

Not applicable

Operating System

Linux 6.18 (Ubuntu container)

Python Version

Python 3.12.3

Additional Context

When cases 1, 3 and 4 are used as an if condition, validate_workflow returns no errors. Case 2 is flagged in a condition, but as unbalanced delimiters and quotes rather than as an escaped quote.

Related: #5060 proposes replacing the hand-written expression parser with a grammar so that invalid input raises. It does not mention this case.

Also reproduced on macOS (Darwin 25.6, arm64) with Python 3.14.4 and uv run python -I repro.py against 27d14e2; the output matches the above.

AI Disclosure

Found and reproduced with Claude Code (model: Claude Opus 4.8), run autonomously under human supervision. The human reporter re-ran every reproduction above before filing.

Lenguaje dominante
Python
Estrellas
149k
Forks
13.4k
Merge medio
1 d 5 h
PR fusionados (30 d)
223

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de github/spec-kit

Todos los issues de github/spec-kit

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.