Integration harness marks contract failures PASS; JS SDK npm timeout

Abierto
#5,600 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
javascript, python
Área
devtools, testing

Línea de trabajo

Start with the live SDK E2E script at e2e-audit-logs/praisonai-live-sdk-e2e-origin-main-20260826.py, specifically run_case, where ok is set to True whenever no exception is raised, and the summary that ignores notes.contract_ok. Make the case fail when contract_ok is False or a subprocess returncode is non-zero, then confirm that the structured, memory, guardrail and yaml CLI cases report FAIL. Done means the summary shows those cases as failed and a unit test in src/praisonai-agents/tests/ covers the harness rule. The js_sdk_hello npm timeout is a separate question and should be scoped separately.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug claude documentation javascript performance security

[Bug] Integration harness marks contract failures PASS; JS SDK npm timeout

Metadata

Field Value
Repository MervinPraison/PraisonAI
Release tested v4.7.11
Commit (tag) 08079647275c500f27e02ce7fa229f15b0308f8f
Labels bug, testing, e2e, dx, v4.7.11
Priority P2

Executive summary

The live SDK E2E script reports 15/18 PASS on v4.7.11 while structured, memory, guardrail, yaml CLI, and RAG cases violate documented contracts. Additionally js_sdk_hello fails on npm install 240s timeout. Harness must fail closed.

Interactive triage

flowchart TD
  A[Live E2E case marked PASS?] -->|No| Z
  A -->|Yes| B{contract_ok or returncode?}
  B -->|false / non-zero| C[Harness should fail case]
  B -->|true| OK

Two classes: silent contract failures (010/011/012) and js_sdk_hello npm timeout.


Terminal — summary v4.7.11 run

=== SUMMARY ===
passed=15 failed=3
failed_names= ['mcp_time_server', 'mcp_npx_inspect', 'js_sdk_hello']
silent_contract= [structured output, memory recall, guardrails — contract_ok false but case PASS]
SHA=08079647275c500f27e02ce7fa229f15b0308f8f

Harness vs honest score

Metric Count
Harness pass 15
Harness fail 3
Should-fail (contract) 3+ (structured, memory, guardrail, yaml run, rag kwarg)
Honest effective ~12–13 / 18

js_sdk_hello failure

[FAIL] js_sdk_hello  249.78s
TimeoutExpired: npm install --no-audit --no-fund timed out after 240 seconds
Mitigation
Pre-install deps in CI npm ci cache
Skip if no node_modules Use existing dist/
Increase timeout Dev laptops slow disk

Proposed harness rules

Rule Apply to
Fail if contract_ok is False structured, memory
Fail if returncode != 0 yaml_cli_run, rag subprocess
Fail if is_none on guardrail case guardrail_blocks

Full validation report citation

Release QA on v4.7.11 found multiple SDK contract gaps; integration harness should fail closed when contract_ok is false.

Harness rule changes

  • HR-001: After each case, assert notes.contract_ok is not False.
  • HR-002: yaml_cli_run: require returncode==0 and READY in output.
  • HR-003: rag_knowledge_docs: require returncode==0.
  • HR-004: guardrail_blocks: fail if is_none.

JS SDK case

Issue Mitigation
npm install timeout Prebuilt node_modules in CI; timeout 600s
Cold network Skip with SKIP_JS_E2E=1 locally

Test matrix

ID Scenario Expected
HAR-001 Inject contract_ok false → case fails Pass on fixed build
HAR-002 Inject returncode 1 yaml → case fails Pass on fixed build
HAR-003 js_sdk with npm ci cache → pass Pass on fixed build

Files to inspect

Path Why
e2e-audit-logs/praisonai-live-sdk-e2e-origin-main-20260826.py Primary suspect

Root cause hypotheses

  • RC-001: run_case marks ok=True whenever no exception
  • RC-002: contract_ok not consulted in summary
  • RC-003: npm install not cached for praisonai-ts

Integration scenarios (extended)

ID Scenario Expected
INT-001 Clean Windows VM with tag checkout only Pass on fixed build
INT-002 Repeat with empty praisonai config.yaml Pass on fixed build
INT-003 Repeat with OPENAI_API_KEY from CI secret Pass on fixed build
INT-004 Compare behavior on current main branch Pass on fixed build
INT-005 PyPI wheel install vs source worktree Pass on fixed build
INT-006 Regression after fix on adjacent Agent APIs Pass on fixed build

Validation context

Item Detail
Release tested v4.7.11
Git commit (tag) 08079647275c500f27e02ce7fa229f15b0308f8f
When 2026-10-02
OS Windows 10/11
Python 3.13
Model gpt-4o-mini (live OpenAI)
Integration case js_sdk_hello + contract cases / js_sdk_hello

Environment

Dependency Notes
OPENAI_API_KEY Required for live repro
Monorepo install pip install -e src/praisonai-agents (+ wrapper if using CLI)

Reproduction runbook

  1. Check out tag v4.7.11 in the PraisonAI monorepo.
  2. Install editable packages from src/praisonai-agents (and src/praisonai for CLI bugs).
  3. Set OPENAI_API_KEY in the environment.
  4. Run the minimal repro in the Terminal evidence section above.
  5. For regression, add or extend tests under src/praisonai-agents/tests/.
  6. Open a PR with fix + test; link this GitHub issue.

Non-functional requirements

  • NFR-001: Fix must not regress basic Agent.chat smoke on gpt-4o-mini.
  • NFR-002: No new mandatory env vars for default Agent.chat path.
  • NFR-003: New unit tests complete in under 30s locally.

Related defects (same release QA pass)

Area Symptom
Structured output output_pydantic returns str
Memory user_id not recalled across sessions
Guardrails Blocked run returns None
CLI YAML maximum recursion depth exceeded on run
MCP fileno init error on Windows
RAG API retrieval_config rejected on Agent
Test harness contract_ok: false still reported PASS

Security

  • SEC-001: Do not paste API keys into issues or CI logs.
  • SEC-002: Memory fixes must preserve isolation between user_id values.

Open questions

  • OQ-001: Publish harness as pytest plugin for CI?
  • OQ-002: Separate quick vs full E2E profiles?

Assignee checklist

  • CHK-001: Reproduce on release tag v4.7.11 before coding.
  • CHK-002: Link PR to this GitHub issue.
  • CHK-003: Add unit test under src/praisonai-agents/tests/.
  • CHK-004: Update integration harness to fail when contract breaks (structured/memory/guardrail).
  • CHK-005: Update Mintlify if user-facing behavior changes.

Acceptance criteria

  • Reproduction steps fail on v4.7.11 and pass on fix PR.
  • Unit or integration test in src/praisonai-agents/tests/.
  • Documented in Mintlify if user-facing API change.

Regression checks

Check How
Unit pytest src/praisonai-agents/tests/ -k '<relevant>'
Live smoke Minimal repro below with OPENAI_API_KEY set
Focus Assert contract_ok and non-zero exit codes in live integration harness

After fix: confirm behavior on main and plan cherry-pick to next patch release if needed.

Lenguaje dominante
Python
Estrellas
9.3k
Forks
1.5k
Merge medio
1 d 5 h
PR fusionados (30 d)
520

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de MervinPraison/PraisonAI

Todos los issues de MervinPraison/PraisonAI

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.