Release gate: require 7/7 live SDK smoke before publish

Đang mở
#5,780 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

  • #5788 của @praisonai-triage-agent — đã đóng, không merge

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
25/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
github-actions, python
Lĩnh vực
ci-cd

Hướng nghiên cứu

Start with the harness the issue names, scripts/live_sdk_validation_v4712.py, and confirm it exits non-zero and prints failed scenario names; scripts/live_sdk_remaining_surfaces.py is the extended set. The gate itself belongs in a new workflow under .github/workflows/, which needs an LLM API key secret, so check with maintainers first. Done when a manual workflow_dispatch run goes green and uploads the summary JSON, and a failing scenario blocks publish. Linked PR 5788 was closed unmerged, so read why before starting.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug claude documentation security

Release gate: require 7/7 live SDK smoke before publish

Type

Epic / Process

Labels

release · qa · sdk-trust


Summary

Dimension Detail
Intent No release candidate ships without seven core live SDK scenarios green
Harness scripts/live_sdk_validation_v4712.py
Extended scripts/live_sdk_remaining_surfaces.py (nightly recommended)
Secret LLM provider API key via CI secret store
Outcome Centralized evidence; failed scenario names visible in CI logs

Problem

Core SDK behavior is still proven mostly by ad hoc developer runs. Without a mandatory gate, packages can publish while memory, tools, structured output, or AGUI surfaces regressed. Failures surface in production integrator repos instead of upstream CI.


Goal

Attach the 7-scenario live harness to the release process: automated workflow and/or non-skippable release checklist item before registry publish.


Release pipeline (target)

┌──────────────┐     RC tag / release branch     ┌─────────────────────┐
│  Maintainer  │ ───────────────────────────────►│  CI: live_sdk job   │
└──────────────┘                                 │  (LLM API key)      │
                                                 └──────────┬──────────┘
                                                            │
                              ┌─────────────────────────────┴─────────────────────────────┐
                              │                                                           │
                         7/7 PASS                                                    ANY FAIL
                              │                                                           │
                              ▼                                                           ▼
                    ┌──────────────────┐                                      ┌──────────────────┐
                    │ Publish packages │                                      │ Block release    │
                    │ + release notes  │                                      │ + open defect    │
                    └──────────────────┘                                      └──────────────────┘

Core scenarios (blocking)

Scenario Input shape Expected outcome Unexpected outcome Flake policy
agent_chat_minimal Plain text user message Non-empty assistant string Timeout / empty Retry once on timeout
structured_output_pydantic Pydantic model on agent Instance of model type Dict instead of model Structured output fixes
single_tool_invocation One registered tool Tool call observed Hallucinated args Tool schema tightening
memory_cross_instance_user_id Two agents, one user_id Second agent recalls first Missed recall Prefetch parity
memory_dict_db_path_string Memory dict with db path Store + read OK Path error Filesystem permissions
agui_sse_live_agent AGUI server + agent SSE events received No events Port conflict
package_version Import agents package Version string present Import failure Packaging layout

Extended scenarios (non-blocking by default)

Scenario Knob Expected Unexpected Notes
stream_chat stream=True Token events None returned as success Stream routing
achat_minimal await achat Completion Loop blocked Async client
handoff Two agents handoff Delegated reply Same agent answers Handoff config
agent_team AgentTeam tasks list Team result TypeError on tasks tasks=list(...)
workflow Workflow definition Run completes Step skip Workflow graph
history_second_turn Two turns same session Uses prior turn Refusal text Prompt wording
json_schema_output Schema constrained output Valid JSON instance Schema violation Provider support

Expected vs actual (process)

Area Expected operating model Common gap today
Release branch Harness green before tag Manual spot checks only
Evidence JSON summary archived in CI Scattered terminal history
Failure triage Named scenario in log Generic exit code 1
Extended surfaces Nightly job Runs only before demos
Dependency fixes Gated with harness Merged without live run

CI job design

Control Setting Rationale Anti-pattern
Trigger workflow_dispatch Manual release manager On every push
Secret LLM provider API key in org secrets Injected to job Missing
Fork PRs No secret access Job skipped, comment Hard fail
Artifact Upload harness JSON 30-day retention No artifact
Retry LLM-sensitive scenarios Max 1 retry Infinite retry
Block exit!=0 Fail job, no publish Warn only
Workflow sketch (implement in repository)
name: live-sdk-smoke
on:
  workflow_dispatch:
  pull_request:
    types: [labeled]
jobs:
  smoke:
    if: github.event.label.name == 'run-live-smoke' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -e src/praisonai-agents
      - run: python scripts/live_sdk_validation_v4712.py
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
      - uses: actions/upload-artifact@v4
        with:
          name: live-sdk-summary
          path: "**/live_sdk*.json"

Terminal evidence — expected

$ python scripts/live_sdk_validation_v4712.py
[1/7] agent_chat_minimal ........ PASS
[2/7] structured_output_pydantic  PASS
[3/7] single_tool_invocation ..... PASS
[4/7] memory_cross_instance_user_id PASS
[5/7] memory_dict_db_path_string . PASS
[6/7] agui_sse_live_agent ....... PASS
[7/7] package_version ............ PASS
exit=0
{"passed": 7, "failed": 0, "total": 7, "release_tag": "main"}

CI tail (expected):

Job smoke completed successfully
Artifact live-sdk-summary uploaded

Terminal evidence — unexpected

$ python scripts/live_sdk_validation_v4712.py
[4/7] memory_cross_instance_user_id FAIL
exit=1
{"passed": 6, "failed": 1, "total": 7, "failures": ["memory_cross_instance_user_id"]}

Missing API key (expected handling — not a product failure):

ERROR: LLM API key not set — skipping live smoke job
Job result: skipped (documented)

Bad handling (must fix):

Error: Authentication failed
exit=1
(no scenario list printed)

Failure mode catalog

Surface Risk Detection Mitigation
chat(stream=None) Wrong branch stream scenario in extended harness Stream routing fix + matrix doc
Memory prefetch drift Scenario 4 fail Core harness Async/sync parity
Structured output Scenario 2 fail Core harness Pydantic coercion fix
AGUI port Scenario 6 fail Core harness Bind retry / doc port
Missing CI secret False red Job logs Skip with explicit reason
Over-aggressive retry Hides flakes Metrics Cap retries at 1

How to fix (engineering checklist)

# Task Owner Done when
1 Add workflow file under .github/workflows/ Platform Dispatches manually
2 Document release manager duty in contributing guide Docs Section linked from README
3 Ensure harness exits non-zero on any failure SDK Unit test on driver
4 Print failure array in JSON SDK Logged in CI
5 Schedule extended harness cron Platform Weekly email on fail
6 Label run-live-smoke documented for maintainers Docs Label exists on repo

Rollout phases

Phase Scope Success metric
A Manual workflow_dispatch only 3 green runs recorded
B Required on release branch tags Zero publish with red harness
C Optional PR label for risky SDK PRs Maintainers use label
D Weekly extended harness Trend dashboard

Acceptance criteria

  • Policy text: 7/7 core required before publish
  • CI or checklist enforces policy without silent bypass
  • Failed scenario names appear in logs and artifact JSON
  • Extended harness scheduled at least weekly

Out of scope

  • Shell-capable sandbox models (separate enablement track)
  • Third-party crew marketplace parity
  • Prebuilt tool catalog count competition

Engineering FAQ

Question Answer Verify with
Who runs harness if CI skipped? Release manager on release branch before tag CONTRIBUTING section
Cost control? workflow_dispatch default; label trigger optional GitHub Actions billing
Fork PRs? Skip job — no org secrets Comment on PR
Which Python? Match lowest supported 3.x in package metadata setup-python matrix
Editable install in CI? pip install -e src/praisonai-agents Job log
Artifact retention? 30 days default upload-artifact Actions settings
Flake on scenario 7? Rare — import/path; do not retry blindly Isolate import test
Extended harness blocking? No by default; nightly alerts only Cron workflow
AGUI flake? Port bind — retry once Matrix note
Memory flake? Fix code not retry policy Scenario 4 logs
History scenario flake? Model refusal — retry once max Extended harness only
Publish with 6/7? Forbidden once policy active Audit tags
Manual waiver? Requires written exception in release notes Manager process
Key rotation? Org secret update only; no commit Secret store
Repro without CI? Same harness script as automation job Identical exit code
Timeout per scenario? Harness should enforce per-scenario ceiling Add if missing
Parallel scenarios? Sequential preferred for log clarity Harness design
JSON schema location? Printed stdout tail + artifact glob Upload step
Red action? Block merge on release branch if job required Branch protection
Green action? Allow publish workflow to proceed Release automation

Operational runbook

Step Action Expected signal If fail
1 Checkout release branch Clean working tree Stash changes
2 Install agents package editable import praisonaiagents ok Fix PYTHONPATH
3 Export LLM API key env Key present in env Stop — no key no run
4 Run core harness script exit 0 Capture JSON failures
5 If 6/7 memory fail Open async prefetch issue Do not tag release
6 If structured fail Open pydantic coercion issue Same
7 Run extended harness optional Log only File separate defects
8 Upload JSON to release ticket Artifact attached Manager visibility
9 Trigger CI smoke workflow Green job Compare CI vs machine
10 Tag only after green Git tag created Revert tag if later red
11 Publish packages Registry shows version Yank if post-publish red
12 Update behavior matrix date Docs PR merged Track drift
13 Announce in changelog Harness mentioned Transparency
14 Schedule weekly extended job Cron enabled Alert on fail
15 Postmortem if waiver used Document in release notes Policy audit

Evidence log template

Run Date Command Expected marker Unexpected marker
RC-1 YYYY-MM-DD live_sdk_validation_v4712.py passed=7 failed>=1
RC-2 YYYY-MM-DD CI workflow_dispatch Job green Red/no artifact
RC-3 YYYY-MM-DD live_sdk_remaining_surfaces.py exit 0 Scenario name fail
RC-4 YYYY-MM-DD Re-run after memory fix scenario 4 PASS 6/7
RC-5 YYYY-MM-DD Re-run after stream fix stream PASS None success
RC-6 YYYY-MM-DD Pre-tag gate Two greens in row Single green only
RC-7 YYYY-MM-DD Post-publish spot check Still 7/7 Regression hotfix

Gate ownership

Gate stage Owner Artifact Hard fail condition
Pre-merge SDK PR Author Optional label job N/A unless required
Release branch Release manager Harness JSON Any core scenario fail
Tag Release manager CI smoke green Red workflow
Publish Automation Changelog link Missing gate waiver
Post-release Quality Weekly extended job Trend >1 fail/week
Ngôn ngữ chính
Python
Star
9.3k
Fork
1.5k
Merge trung bình
1 ngày 5 giờ
Pull request đã merge (30 ngày)
520

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của MervinPraison/PraisonAI

Tất cả issue của MervinPraison/PraisonAI

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.