Release gate: require 7/7 live SDK smoke before publish
Maintainer thường phản hồi trong vòng 1 ngày
Chưa có ai nhận issue này.
- #5788 của @praisonai-triage-agent — đã đóng, không merge
Đánh giá
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức phù hợp với người mới
- 25/100
Hướng nghiên cứu
Start with the harness the issue names, scripts/live_sdk_validation_v4712.py, and confirm it exits non-zero and prints failed scenario names; scripts/live_sdk_remaining_surfaces.py is the extended set. The gate itself belongs in a new workflow under .github/workflows/, which needs an LLM API key secret, so check with maintainers first. Done when a manual workflow_dispatch run goes green and uploads the summary JSON, and a failing scenario blocks publish. Linked PR 5788 was closed unmerged, so read why before starting.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
Release gate: require 7/7 live SDK smoke before publish
Type
Epic / Process
Labels
release · qa · sdk-trust
Summary
| Dimension | Detail |
|---|---|
| Intent | No release candidate ships without seven core live SDK scenarios green |
| Harness | scripts/live_sdk_validation_v4712.py |
| Extended | scripts/live_sdk_remaining_surfaces.py (nightly recommended) |
| Secret | LLM provider API key via CI secret store |
| Outcome | Centralized evidence; failed scenario names visible in CI logs |
Problem
Core SDK behavior is still proven mostly by ad hoc developer runs. Without a mandatory gate, packages can publish while memory, tools, structured output, or AGUI surfaces regressed. Failures surface in production integrator repos instead of upstream CI.
Goal
Attach the 7-scenario live harness to the release process: automated workflow and/or non-skippable release checklist item before registry publish.
Release pipeline (target)
┌──────────────┐ RC tag / release branch ┌─────────────────────┐
│ Maintainer │ ───────────────────────────────►│ CI: live_sdk job │
└──────────────┘ │ (LLM API key) │
└──────────┬──────────┘
│
┌─────────────────────────────┴─────────────────────────────┐
│ │
7/7 PASS ANY FAIL
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Publish packages │ │ Block release │
│ + release notes │ │ + open defect │
└──────────────────┘ └──────────────────┘
Core scenarios (blocking)
| Scenario | Input shape | Expected outcome | Unexpected outcome | Flake policy |
|---|---|---|---|---|
| agent_chat_minimal | Plain text user message | Non-empty assistant string | Timeout / empty | Retry once on timeout |
| structured_output_pydantic | Pydantic model on agent | Instance of model type | Dict instead of model | Structured output fixes |
| single_tool_invocation | One registered tool | Tool call observed | Hallucinated args | Tool schema tightening |
| memory_cross_instance_user_id | Two agents, one user_id | Second agent recalls first | Missed recall | Prefetch parity |
| memory_dict_db_path_string | Memory dict with db path | Store + read OK | Path error | Filesystem permissions |
| agui_sse_live_agent | AGUI server + agent | SSE events received | No events | Port conflict |
| package_version | Import agents package | Version string present | Import failure | Packaging layout |
Extended scenarios (non-blocking by default)
| Scenario | Knob | Expected | Unexpected | Notes |
|---|---|---|---|---|
| stream_chat | stream=True | Token events | None returned as success | Stream routing |
| achat_minimal | await achat | Completion | Loop blocked | Async client |
| handoff | Two agents handoff | Delegated reply | Same agent answers | Handoff config |
| agent_team | AgentTeam tasks list | Team result | TypeError on tasks | tasks=list(...) |
| workflow | Workflow definition | Run completes | Step skip | Workflow graph |
| history_second_turn | Two turns same session | Uses prior turn | Refusal text | Prompt wording |
| json_schema_output | Schema constrained output | Valid JSON instance | Schema violation | Provider support |
Expected vs actual (process)
| Area | Expected operating model | Common gap today |
|---|---|---|
| Release branch | Harness green before tag | Manual spot checks only |
| Evidence | JSON summary archived in CI | Scattered terminal history |
| Failure triage | Named scenario in log | Generic exit code 1 |
| Extended surfaces | Nightly job | Runs only before demos |
| Dependency fixes | Gated with harness | Merged without live run |
CI job design
| Control | Setting | Rationale | Anti-pattern |
|---|---|---|---|
| Trigger | workflow_dispatch | Manual release manager | On every push |
| Secret | LLM provider API key in org secrets | Injected to job | Missing |
| Fork PRs | No secret access | Job skipped, comment | Hard fail |
| Artifact | Upload harness JSON | 30-day retention | No artifact |
| Retry | LLM-sensitive scenarios | Max 1 retry | Infinite retry |
| Block | exit!=0 | Fail job, no publish | Warn only |
Workflow sketch (implement in repository)
name: live-sdk-smoke
on:
workflow_dispatch:
pull_request:
types: [labeled]
jobs:
smoke:
if: github.event.label.name == 'run-live-smoke' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -e src/praisonai-agents
- run: python scripts/live_sdk_validation_v4712.py
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
- uses: actions/upload-artifact@v4
with:
name: live-sdk-summary
path: "**/live_sdk*.json"
Terminal evidence — expected
$ python scripts/live_sdk_validation_v4712.py
[1/7] agent_chat_minimal ........ PASS
[2/7] structured_output_pydantic PASS
[3/7] single_tool_invocation ..... PASS
[4/7] memory_cross_instance_user_id PASS
[5/7] memory_dict_db_path_string . PASS
[6/7] agui_sse_live_agent ....... PASS
[7/7] package_version ............ PASS
exit=0
{"passed": 7, "failed": 0, "total": 7, "release_tag": "main"}
CI tail (expected):
Job smoke completed successfully
Artifact live-sdk-summary uploaded
Terminal evidence — unexpected
$ python scripts/live_sdk_validation_v4712.py
[4/7] memory_cross_instance_user_id FAIL
exit=1
{"passed": 6, "failed": 1, "total": 7, "failures": ["memory_cross_instance_user_id"]}
Missing API key (expected handling — not a product failure):
ERROR: LLM API key not set — skipping live smoke job
Job result: skipped (documented)
Bad handling (must fix):
Error: Authentication failed
exit=1
(no scenario list printed)
Failure mode catalog
| Surface | Risk | Detection | Mitigation |
|---|---|---|---|
chat(stream=None) |
Wrong branch | stream scenario in extended harness | Stream routing fix + matrix doc |
| Memory prefetch drift | Scenario 4 fail | Core harness | Async/sync parity |
| Structured output | Scenario 2 fail | Core harness | Pydantic coercion fix |
| AGUI port | Scenario 6 fail | Core harness | Bind retry / doc port |
| Missing CI secret | False red | Job logs | Skip with explicit reason |
| Over-aggressive retry | Hides flakes | Metrics | Cap retries at 1 |
How to fix (engineering checklist)
| # | Task | Owner | Done when |
|---|---|---|---|
| 1 | Add workflow file under .github/workflows/ |
Platform | Dispatches manually |
| 2 | Document release manager duty in contributing guide | Docs | Section linked from README |
| 3 | Ensure harness exits non-zero on any failure | SDK | Unit test on driver |
| 4 | Print failure array in JSON | SDK | Logged in CI |
| 5 | Schedule extended harness cron | Platform | Weekly email on fail |
| 6 | Label run-live-smoke documented for maintainers |
Docs | Label exists on repo |
Rollout phases
| Phase | Scope | Success metric |
|---|---|---|
| A | Manual workflow_dispatch only | 3 green runs recorded |
| B | Required on release branch tags | Zero publish with red harness |
| C | Optional PR label for risky SDK PRs | Maintainers use label |
| D | Weekly extended harness | Trend dashboard |
Acceptance criteria
- Policy text: 7/7 core required before publish
- CI or checklist enforces policy without silent bypass
- Failed scenario names appear in logs and artifact JSON
- Extended harness scheduled at least weekly
Out of scope
- Shell-capable sandbox models (separate enablement track)
- Third-party crew marketplace parity
- Prebuilt tool catalog count competition
Engineering FAQ
| Question | Answer | Verify with |
|---|---|---|
| Who runs harness if CI skipped? | Release manager on release branch before tag | CONTRIBUTING section |
| Cost control? | workflow_dispatch default; label trigger optional | GitHub Actions billing |
| Fork PRs? | Skip job — no org secrets | Comment on PR |
| Which Python? | Match lowest supported 3.x in package metadata | setup-python matrix |
| Editable install in CI? | pip install -e src/praisonai-agents | Job log |
| Artifact retention? | 30 days default upload-artifact | Actions settings |
| Flake on scenario 7? | Rare — import/path; do not retry blindly | Isolate import test |
| Extended harness blocking? | No by default; nightly alerts only | Cron workflow |
| AGUI flake? | Port bind — retry once | Matrix note |
| Memory flake? | Fix code not retry policy | Scenario 4 logs |
| History scenario flake? | Model refusal — retry once max | Extended harness only |
| Publish with 6/7? | Forbidden once policy active | Audit tags |
| Manual waiver? | Requires written exception in release notes | Manager process |
| Key rotation? | Org secret update only; no commit | Secret store |
| Repro without CI? | Same harness script as automation job | Identical exit code |
| Timeout per scenario? | Harness should enforce per-scenario ceiling | Add if missing |
| Parallel scenarios? | Sequential preferred for log clarity | Harness design |
| JSON schema location? | Printed stdout tail + artifact glob | Upload step |
| Red action? | Block merge on release branch if job required | Branch protection |
| Green action? | Allow publish workflow to proceed | Release automation |
Operational runbook
| Step | Action | Expected signal | If fail |
|---|---|---|---|
| 1 | Checkout release branch | Clean working tree | Stash changes |
| 2 | Install agents package editable | import praisonaiagents ok | Fix PYTHONPATH |
| 3 | Export LLM API key env | Key present in env | Stop — no key no run |
| 4 | Run core harness script | exit 0 | Capture JSON failures |
| 5 | If 6/7 memory fail | Open async prefetch issue | Do not tag release |
| 6 | If structured fail | Open pydantic coercion issue | Same |
| 7 | Run extended harness optional | Log only | File separate defects |
| 8 | Upload JSON to release ticket | Artifact attached | Manager visibility |
| 9 | Trigger CI smoke workflow | Green job | Compare CI vs machine |
| 10 | Tag only after green | Git tag created | Revert tag if later red |
| 11 | Publish packages | Registry shows version | Yank if post-publish red |
| 12 | Update behavior matrix date | Docs PR merged | Track drift |
| 13 | Announce in changelog | Harness mentioned | Transparency |
| 14 | Schedule weekly extended job | Cron enabled | Alert on fail |
| 15 | Postmortem if waiver used | Document in release notes | Policy audit |
Evidence log template
| Run | Date | Command | Expected marker | Unexpected marker |
|---|---|---|---|---|
| RC-1 | YYYY-MM-DD | live_sdk_validation_v4712.py | passed=7 | failed>=1 |
| RC-2 | YYYY-MM-DD | CI workflow_dispatch | Job green | Red/no artifact |
| RC-3 | YYYY-MM-DD | live_sdk_remaining_surfaces.py | exit 0 | Scenario name fail |
| RC-4 | YYYY-MM-DD | Re-run after memory fix | scenario 4 PASS | 6/7 |
| RC-5 | YYYY-MM-DD | Re-run after stream fix | stream PASS | None success |
| RC-6 | YYYY-MM-DD | Pre-tag gate | Two greens in row | Single green only |
| RC-7 | YYYY-MM-DD | Post-publish spot check | Still 7/7 | Regression hotfix |
Gate ownership
| Gate stage | Owner | Artifact | Hard fail condition |
|---|---|---|---|
| Pre-merge SDK PR | Author | Optional label job | N/A unless required |
| Release branch | Release manager | Harness JSON | Any core scenario fail |
| Tag | Release manager | CI smoke green | Red workflow |
| Publish | Automation | Changelog link | Missing gate waiver |
| Post-release | Quality | Weekly extended job | Trend >1 fail/week |
- Ngôn ngữ chính
- Python
- Star
- 9.3k
- Fork
- 1.5k
- Merge trung bình
- 1 ngày 5 giờ
- Pull request đã merge (30 ngày)
- 520
Chuẩn bị môi trường
Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của MervinPraison/PraisonAI
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
MervinPraison/PraisonAI#6710 · 1 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
claude
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 74/100
MervinPraison/PraisonAI#6707 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
bug claude
Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 90/100
MervinPraison/PraisonAI#6671 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Compaction budget regression fixture remains below budget with accurate tokenizationCó thể làm lại được Pull request cho issue này đã bị đóng mà không được merge. Đang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 62/100
MervinPraison/PraisonAI#5418 · 3 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
bug claude documentation
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 70/100
MervinPraison/PraisonAI#5341 · 3 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
Tất cả issue của MervinPraison/PraisonAI
Issue tương tự
-
needs-triage
Độ khó 1/5 1-3 giờ Mức phù hợp với người mới 85/100
gke-labs/kube-agents#6620 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
bug
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 75/100
open-webui/open-webui#34719 · 2 bình luận ·
Maintainer thường phản hồi trong vòng 1 ngày
-
cognition
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
Maintainer thường phản hồi trong vòng 1 ngày
-
One non-object JSON record makes the session catalog exitCó thể đã có người làm @he-yufeng đã nhận hôm nay. Đang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100
Maintainer thường phản hồi trong vòng 1 ngày
-
[Suggestion]: Geonode MCP ServerĐang mở
Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 78/100
collabnix/awesome-mcp-lists#433 ·
Maintainer thường phản hồi trong vòng 4 ngày