Integration harness marks contract failures PASS; JS SDK npm timeout
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Aptitud para principiantes
- 45/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Tranquilo
- Stack tecnológico
- javascript, python
Línea de trabajo
Start with the live SDK E2E script at e2e-audit-logs/praisonai-live-sdk-e2e-origin-main-20260826.py, specifically run_case, where ok is set to True whenever no exception is raised, and the summary that ignores notes.contract_ok. Make the case fail when contract_ok is False or a subprocess returncode is non-zero, then confirm that the structured, memory, guardrail and yaml CLI cases report FAIL. Done means the summary shows those cases as failed and a unit test in src/praisonai-agents/tests/ covers the harness rule. The js_sdk_hello npm timeout is a separate question and should be scoped separately.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
[Bug] Integration harness marks contract failures PASS; JS SDK npm timeout
Metadata
| Field | Value |
|---|---|
| Repository | MervinPraison/PraisonAI |
| Release tested | v4.7.11 |
| Commit (tag) | 08079647275c500f27e02ce7fa229f15b0308f8f |
| Labels | bug, testing, e2e, dx, v4.7.11 |
| Priority | P2 |
Executive summary
The live SDK E2E script reports 15/18 PASS on v4.7.11 while structured, memory, guardrail, yaml CLI, and RAG cases violate documented contracts. Additionally js_sdk_hello fails on npm install 240s timeout. Harness must fail closed.
Interactive triage
flowchart TD
A[Live E2E case marked PASS?] -->|No| Z
A -->|Yes| B{contract_ok or returncode?}
B -->|false / non-zero| C[Harness should fail case]
B -->|true| OK
Two classes: silent contract failures (010/011/012) and js_sdk_hello npm timeout.
Terminal — summary v4.7.11 run
=== SUMMARY ===
passed=15 failed=3
failed_names= ['mcp_time_server', 'mcp_npx_inspect', 'js_sdk_hello']
silent_contract= [structured output, memory recall, guardrails — contract_ok false but case PASS]
SHA=08079647275c500f27e02ce7fa229f15b0308f8f
Harness vs honest score
| Metric | Count |
|---|---|
| Harness pass | 15 |
| Harness fail | 3 |
| Should-fail (contract) | 3+ (structured, memory, guardrail, yaml run, rag kwarg) |
| Honest effective | ~12–13 / 18 |
js_sdk_hello failure
[FAIL] js_sdk_hello 249.78s
TimeoutExpired: npm install --no-audit --no-fund timed out after 240 seconds
| Mitigation | |
|---|---|
| Pre-install deps in CI | npm ci cache |
| Skip if no node_modules | Use existing dist/ |
| Increase timeout | Dev laptops slow disk |
Proposed harness rules
| Rule | Apply to |
|---|---|
Fail if contract_ok is False |
structured, memory |
Fail if returncode != 0 |
yaml_cli_run, rag subprocess |
Fail if is_none on guardrail case |
guardrail_blocks |
Full validation report citation
Release QA on v4.7.11 found multiple SDK contract gaps; integration harness should fail closed when contract_ok is false.
Harness rule changes
- HR-001: After each case, assert notes.contract_ok is not False.
- HR-002: yaml_cli_run: require returncode==0 and READY in output.
- HR-003: rag_knowledge_docs: require returncode==0.
- HR-004: guardrail_blocks: fail if is_none.
JS SDK case
| Issue | Mitigation |
|---|---|
| npm install timeout | Prebuilt node_modules in CI; timeout 600s |
| Cold network | Skip with SKIP_JS_E2E=1 locally |
Test matrix
| ID | Scenario | Expected |
|---|---|---|
| HAR-001 | Inject contract_ok false → case fails | Pass on fixed build |
| HAR-002 | Inject returncode 1 yaml → case fails | Pass on fixed build |
| HAR-003 | js_sdk with npm ci cache → pass | Pass on fixed build |
Files to inspect
| Path | Why |
|---|---|
e2e-audit-logs/praisonai-live-sdk-e2e-origin-main-20260826.py |
Primary suspect |
Root cause hypotheses
- RC-001: run_case marks ok=True whenever no exception
- RC-002: contract_ok not consulted in summary
- RC-003: npm install not cached for praisonai-ts
Integration scenarios (extended)
| ID | Scenario | Expected |
|---|---|---|
| INT-001 | Clean Windows VM with tag checkout only | Pass on fixed build |
| INT-002 | Repeat with empty praisonai config.yaml | Pass on fixed build |
| INT-003 | Repeat with OPENAI_API_KEY from CI secret | Pass on fixed build |
| INT-004 | Compare behavior on current main branch | Pass on fixed build |
| INT-005 | PyPI wheel install vs source worktree | Pass on fixed build |
| INT-006 | Regression after fix on adjacent Agent APIs | Pass on fixed build |
Validation context
| Item | Detail |
|---|---|
| Release tested | v4.7.11 |
| Git commit (tag) | 08079647275c500f27e02ce7fa229f15b0308f8f |
| When | 2026-10-02 |
| OS | Windows 10/11 |
| Python | 3.13 |
| Model | gpt-4o-mini (live OpenAI) |
| Integration case | js_sdk_hello + contract cases / js_sdk_hello |
Environment
| Dependency | Notes |
|---|---|
OPENAI_API_KEY |
Required for live repro |
| Monorepo install | pip install -e src/praisonai-agents (+ wrapper if using CLI) |
Reproduction runbook
- Check out tag v4.7.11 in the PraisonAI monorepo.
- Install editable packages from
src/praisonai-agents(andsrc/praisonaifor CLI bugs). - Set
OPENAI_API_KEYin the environment. - Run the minimal repro in the Terminal evidence section above.
- For regression, add or extend tests under
src/praisonai-agents/tests/. - Open a PR with fix + test; link this GitHub issue.
Non-functional requirements
- NFR-001: Fix must not regress basic Agent.chat smoke on gpt-4o-mini.
- NFR-002: No new mandatory env vars for default Agent.chat path.
- NFR-003: New unit tests complete in under 30s locally.
Related defects (same release QA pass)
| Area | Symptom |
|---|---|
| Structured output | output_pydantic returns str |
| Memory | user_id not recalled across sessions |
| Guardrails | Blocked run returns None |
| CLI YAML | maximum recursion depth exceeded on run |
| MCP | fileno init error on Windows |
| RAG API | retrieval_config rejected on Agent |
| Test harness | contract_ok: false still reported PASS |
Security
- SEC-001: Do not paste API keys into issues or CI logs.
- SEC-002: Memory fixes must preserve isolation between
user_idvalues.
Open questions
- OQ-001: Publish harness as pytest plugin for CI?
- OQ-002: Separate quick vs full E2E profiles?
Assignee checklist
- CHK-001: Reproduce on release tag v4.7.11 before coding.
- CHK-002: Link PR to this GitHub issue.
- CHK-003: Add unit test under
src/praisonai-agents/tests/. - CHK-004: Update integration harness to fail when contract breaks (structured/memory/guardrail).
- CHK-005: Update Mintlify if user-facing behavior changes.
Acceptance criteria
- Reproduction steps fail on v4.7.11 and pass on fix PR.
- Unit or integration test in
src/praisonai-agents/tests/. - Documented in Mintlify if user-facing API change.
Regression checks
| Check | How |
|---|---|
| Unit | pytest src/praisonai-agents/tests/ -k '<relevant>' |
| Live smoke | Minimal repro below with OPENAI_API_KEY set |
| Focus | Assert contract_ok and non-zero exit codes in live integration harness |
After fix: confirm behavior on main and plan cherry-pick to next patch release if needed.
- Lenguaje dominante
- Python
- Estrellas
- 9.3k
- Forks
- 1.5k
- Merge medio
- 1 d 5 h
- PR fusionados (30 d)
- 520
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de MervinPraison/PraisonAI
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
MervinPraison/PraisonAI#6710 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
claude
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
MervinPraison/PraisonAI#6707 ·
Los mantenedores suelen responder en 1 día
-
bug claude
Dificultad 1/5 Menos de una hora Aptitud para principiantes 90/100
MervinPraison/PraisonAI#6671 ·
Los mantenedores suelen responder en 1 día
-
Compaction budget regression fixture remains below budget with accurate tokenizationQuizá libre de nuevo Un pull request para esta issue se cerró sin fusionarse. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
MervinPraison/PraisonAI#5418 · 3 comentarios ·
Los mantenedores suelen responder en 1 día
-
bug claude documentation
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
MervinPraison/PraisonAI#5341 · 3 comentarios ·
Los mantenedores suelen responder en 1 día
Todos los issues de MervinPraison/PraisonAI
Issues similares
-
cognition
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 1 día
-
[Suggestion]: Geonode MCP ServerAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 78/100
collabnix/awesome-mcp-lists#433 ·
Los mantenedores suelen responder en 4 días
-
CrossEntropyFilter raises on zero probabilities and single-class resultsPosiblemente ocupada @NopeCharon la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
datajoint/datajoint-python#1703 ·
Los mantenedores suelen responder en 6 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Archmonger/ServeStatic#129 ·