Release gate: require 7/7 live SDK smoke before publish
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
- #5788 @praisonai-triage-agent による — マージされずにクローズ
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 25/100
調査の方向性
Start with the harness the issue names, scripts/live_sdk_validation_v4712.py, and confirm it exits non-zero and prints failed scenario names; scripts/live_sdk_remaining_surfaces.py is the extended set. The gate itself belongs in a new workflow under .github/workflows/, which needs an LLM API key secret, so check with maintainers first. Done when a manual workflow_dispatch run goes green and uploads the summary JSON, and a failing scenario blocks publish. Linked PR 5788 was closed unmerged, so read why before starting.
索引モデルが issue の本文から書いたものです。
説明
Release gate: require 7/7 live SDK smoke before publish
Type
Epic / Process
Labels
release · qa · sdk-trust
Summary
| Dimension | Detail |
|---|---|
| Intent | No release candidate ships without seven core live SDK scenarios green |
| Harness | scripts/live_sdk_validation_v4712.py |
| Extended | scripts/live_sdk_remaining_surfaces.py (nightly recommended) |
| Secret | LLM provider API key via CI secret store |
| Outcome | Centralized evidence; failed scenario names visible in CI logs |
Problem
Core SDK behavior is still proven mostly by ad hoc developer runs. Without a mandatory gate, packages can publish while memory, tools, structured output, or AGUI surfaces regressed. Failures surface in production integrator repos instead of upstream CI.
Goal
Attach the 7-scenario live harness to the release process: automated workflow and/or non-skippable release checklist item before registry publish.
Release pipeline (target)
┌──────────────┐ RC tag / release branch ┌─────────────────────┐
│ Maintainer │ ───────────────────────────────►│ CI: live_sdk job │
└──────────────┘ │ (LLM API key) │
└──────────┬──────────┘
│
┌─────────────────────────────┴─────────────────────────────┐
│ │
7/7 PASS ANY FAIL
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ Publish packages │ │ Block release │
│ + release notes │ │ + open defect │
└──────────────────┘ └──────────────────┘
Core scenarios (blocking)
| Scenario | Input shape | Expected outcome | Unexpected outcome | Flake policy |
|---|---|---|---|---|
| agent_chat_minimal | Plain text user message | Non-empty assistant string | Timeout / empty | Retry once on timeout |
| structured_output_pydantic | Pydantic model on agent | Instance of model type | Dict instead of model | Structured output fixes |
| single_tool_invocation | One registered tool | Tool call observed | Hallucinated args | Tool schema tightening |
| memory_cross_instance_user_id | Two agents, one user_id | Second agent recalls first | Missed recall | Prefetch parity |
| memory_dict_db_path_string | Memory dict with db path | Store + read OK | Path error | Filesystem permissions |
| agui_sse_live_agent | AGUI server + agent | SSE events received | No events | Port conflict |
| package_version | Import agents package | Version string present | Import failure | Packaging layout |
Extended scenarios (non-blocking by default)
| Scenario | Knob | Expected | Unexpected | Notes |
|---|---|---|---|---|
| stream_chat | stream=True | Token events | None returned as success | Stream routing |
| achat_minimal | await achat | Completion | Loop blocked | Async client |
| handoff | Two agents handoff | Delegated reply | Same agent answers | Handoff config |
| agent_team | AgentTeam tasks list | Team result | TypeError on tasks | tasks=list(...) |
| workflow | Workflow definition | Run completes | Step skip | Workflow graph |
| history_second_turn | Two turns same session | Uses prior turn | Refusal text | Prompt wording |
| json_schema_output | Schema constrained output | Valid JSON instance | Schema violation | Provider support |
Expected vs actual (process)
| Area | Expected operating model | Common gap today |
|---|---|---|
| Release branch | Harness green before tag | Manual spot checks only |
| Evidence | JSON summary archived in CI | Scattered terminal history |
| Failure triage | Named scenario in log | Generic exit code 1 |
| Extended surfaces | Nightly job | Runs only before demos |
| Dependency fixes | Gated with harness | Merged without live run |
CI job design
| Control | Setting | Rationale | Anti-pattern |
|---|---|---|---|
| Trigger | workflow_dispatch | Manual release manager | On every push |
| Secret | LLM provider API key in org secrets | Injected to job | Missing |
| Fork PRs | No secret access | Job skipped, comment | Hard fail |
| Artifact | Upload harness JSON | 30-day retention | No artifact |
| Retry | LLM-sensitive scenarios | Max 1 retry | Infinite retry |
| Block | exit!=0 | Fail job, no publish | Warn only |
Workflow sketch (implement in repository)
name: live-sdk-smoke
on:
workflow_dispatch:
pull_request:
types: [labeled]
jobs:
smoke:
if: github.event.label.name == 'run-live-smoke' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -e src/praisonai-agents
- run: python scripts/live_sdk_validation_v4712.py
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
- uses: actions/upload-artifact@v4
with:
name: live-sdk-summary
path: "**/live_sdk*.json"
Terminal evidence — expected
$ python scripts/live_sdk_validation_v4712.py
[1/7] agent_chat_minimal ........ PASS
[2/7] structured_output_pydantic PASS
[3/7] single_tool_invocation ..... PASS
[4/7] memory_cross_instance_user_id PASS
[5/7] memory_dict_db_path_string . PASS
[6/7] agui_sse_live_agent ....... PASS
[7/7] package_version ............ PASS
exit=0
{"passed": 7, "failed": 0, "total": 7, "release_tag": "main"}
CI tail (expected):
Job smoke completed successfully
Artifact live-sdk-summary uploaded
Terminal evidence — unexpected
$ python scripts/live_sdk_validation_v4712.py
[4/7] memory_cross_instance_user_id FAIL
exit=1
{"passed": 6, "failed": 1, "total": 7, "failures": ["memory_cross_instance_user_id"]}
Missing API key (expected handling — not a product failure):
ERROR: LLM API key not set — skipping live smoke job
Job result: skipped (documented)
Bad handling (must fix):
Error: Authentication failed
exit=1
(no scenario list printed)
Failure mode catalog
| Surface | Risk | Detection | Mitigation |
|---|---|---|---|
chat(stream=None) |
Wrong branch | stream scenario in extended harness | Stream routing fix + matrix doc |
| Memory prefetch drift | Scenario 4 fail | Core harness | Async/sync parity |
| Structured output | Scenario 2 fail | Core harness | Pydantic coercion fix |
| AGUI port | Scenario 6 fail | Core harness | Bind retry / doc port |
| Missing CI secret | False red | Job logs | Skip with explicit reason |
| Over-aggressive retry | Hides flakes | Metrics | Cap retries at 1 |
How to fix (engineering checklist)
| # | Task | Owner | Done when |
|---|---|---|---|
| 1 | Add workflow file under .github/workflows/ |
Platform | Dispatches manually |
| 2 | Document release manager duty in contributing guide | Docs | Section linked from README |
| 3 | Ensure harness exits non-zero on any failure | SDK | Unit test on driver |
| 4 | Print failure array in JSON | SDK | Logged in CI |
| 5 | Schedule extended harness cron | Platform | Weekly email on fail |
| 6 | Label run-live-smoke documented for maintainers |
Docs | Label exists on repo |
Rollout phases
| Phase | Scope | Success metric |
|---|---|---|
| A | Manual workflow_dispatch only | 3 green runs recorded |
| B | Required on release branch tags | Zero publish with red harness |
| C | Optional PR label for risky SDK PRs | Maintainers use label |
| D | Weekly extended harness | Trend dashboard |
Acceptance criteria
- Policy text: 7/7 core required before publish
- CI or checklist enforces policy without silent bypass
- Failed scenario names appear in logs and artifact JSON
- Extended harness scheduled at least weekly
Out of scope
- Shell-capable sandbox models (separate enablement track)
- Third-party crew marketplace parity
- Prebuilt tool catalog count competition
Engineering FAQ
| Question | Answer | Verify with |
|---|---|---|
| Who runs harness if CI skipped? | Release manager on release branch before tag | CONTRIBUTING section |
| Cost control? | workflow_dispatch default; label trigger optional | GitHub Actions billing |
| Fork PRs? | Skip job — no org secrets | Comment on PR |
| Which Python? | Match lowest supported 3.x in package metadata | setup-python matrix |
| Editable install in CI? | pip install -e src/praisonai-agents | Job log |
| Artifact retention? | 30 days default upload-artifact | Actions settings |
| Flake on scenario 7? | Rare — import/path; do not retry blindly | Isolate import test |
| Extended harness blocking? | No by default; nightly alerts only | Cron workflow |
| AGUI flake? | Port bind — retry once | Matrix note |
| Memory flake? | Fix code not retry policy | Scenario 4 logs |
| History scenario flake? | Model refusal — retry once max | Extended harness only |
| Publish with 6/7? | Forbidden once policy active | Audit tags |
| Manual waiver? | Requires written exception in release notes | Manager process |
| Key rotation? | Org secret update only; no commit | Secret store |
| Repro without CI? | Same harness script as automation job | Identical exit code |
| Timeout per scenario? | Harness should enforce per-scenario ceiling | Add if missing |
| Parallel scenarios? | Sequential preferred for log clarity | Harness design |
| JSON schema location? | Printed stdout tail + artifact glob | Upload step |
| Red action? | Block merge on release branch if job required | Branch protection |
| Green action? | Allow publish workflow to proceed | Release automation |
Operational runbook
| Step | Action | Expected signal | If fail |
|---|---|---|---|
| 1 | Checkout release branch | Clean working tree | Stash changes |
| 2 | Install agents package editable | import praisonaiagents ok | Fix PYTHONPATH |
| 3 | Export LLM API key env | Key present in env | Stop — no key no run |
| 4 | Run core harness script | exit 0 | Capture JSON failures |
| 5 | If 6/7 memory fail | Open async prefetch issue | Do not tag release |
| 6 | If structured fail | Open pydantic coercion issue | Same |
| 7 | Run extended harness optional | Log only | File separate defects |
| 8 | Upload JSON to release ticket | Artifact attached | Manager visibility |
| 9 | Trigger CI smoke workflow | Green job | Compare CI vs machine |
| 10 | Tag only after green | Git tag created | Revert tag if later red |
| 11 | Publish packages | Registry shows version | Yank if post-publish red |
| 12 | Update behavior matrix date | Docs PR merged | Track drift |
| 13 | Announce in changelog | Harness mentioned | Transparency |
| 14 | Schedule weekly extended job | Cron enabled | Alert on fail |
| 15 | Postmortem if waiver used | Document in release notes | Policy audit |
Evidence log template
| Run | Date | Command | Expected marker | Unexpected marker |
|---|---|---|---|---|
| RC-1 | YYYY-MM-DD | live_sdk_validation_v4712.py | passed=7 | failed>=1 |
| RC-2 | YYYY-MM-DD | CI workflow_dispatch | Job green | Red/no artifact |
| RC-3 | YYYY-MM-DD | live_sdk_remaining_surfaces.py | exit 0 | Scenario name fail |
| RC-4 | YYYY-MM-DD | Re-run after memory fix | scenario 4 PASS | 6/7 |
| RC-5 | YYYY-MM-DD | Re-run after stream fix | stream PASS | None success |
| RC-6 | YYYY-MM-DD | Pre-tag gate | Two greens in row | Single green only |
| RC-7 | YYYY-MM-DD | Post-publish spot check | Still 7/7 | Regression hotfix |
Gate ownership
| Gate stage | Owner | Artifact | Hard fail condition |
|---|---|---|---|
| Pre-merge SDK PR | Author | Optional label job | N/A unless required |
| Release branch | Release manager | Harness JSON | Any core scenario fail |
| Tag | Release manager | CI smoke green | Red workflow |
| Publish | Automation | Changelog link | Missing gate waiver |
| Post-release | Quality | Weekly extended job | Trend >1 fail/week |
- 主要言語
- Python
- スター
- 9.3k
- フォーク
- 1.5k
- 平均マージ
- 1日 5時間
- マージ済み PR(30日)
- 520
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
MervinPraison/PraisonAI のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
MervinPraison/PraisonAI#6710 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
claude
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
MervinPraison/PraisonAI#6707 ·
メンテナーはふだん 1 日以内に返信
-
bug claude
難易度 1/5 1時間未満 初心者へのやさしさ 90/100
MervinPraison/PraisonAI#6671 ·
メンテナーはふだん 1 日以内に返信
-
Compaction budget regression fixture remains below budget with accurate tokenization再び着手できるかも このイシューのプルリクエストはマージされずにクローズされました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
MervinPraison/PraisonAI#5418 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
-
bug claude documentation
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
MervinPraison/PraisonAI#5341 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
MervinPraison/PraisonAI の issue をすべて見る
似ている issue
-
cognition
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 78/100
collabnix/awesome-mcp-lists#433 ·
メンテナーはふだん 4 日以内に返信
-
CrossEntropyFilter raises on zero probabilities and single-class results対応中かも @NopeCharon が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
datajoint/datajoint-python#1703 ·
メンテナーはふだん 6 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
Archmonger/ServeStatic#129 ·