[epic][video] Cost Picker parity — multi-provider video generation
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 18/100
- issue の種類
- 機能追加
- 明瞭さ
- 説明が足りない
- 活発さ
- 活発
- 技術スタック
- python
- 領域
- ai-infra-agents, api
調査の方向性
This is an epic with no single change to make. Start by reading src/praisonai-agents/praisonaiagents/agent/video_agent.py and src/praisonai/praisonai/capabilities/videos.py to see how VideoAgent.generate() currently routes through LiteLLM. Done means one child deliverable (such as the provider registry, #02) is agreed with maintainers and merged with mocked HTTP tests; the epic itself is not a starter task.
索引モデルが issue の本文から書いたものです。
説明
Executive summary
The Video Model Cost Picker is a decision support tool for music-video and short-clip production: it compares 30+ routes spanning zero-cost ffmpeg/ComfyUI workflows, Replicate predictions, fal.ai queues, OpenRouter video jobs, direct Vidu/Alibaba APIs, and first-party OpenAI/Google/Runway APIs.
PraisonAI Python VideoAgent currently generates video only through LiteLLM, which implements five upstream families: OpenAI Sora, Azure Sora, Gemini Veo 3.1, Vertex Veo 3.1, RunwayML. Every other picker row is unsupported in code. OpenRouter integration today is chat-only (praisonai.llm.gateways.OpenRouterProvider). Replicate is LLM examples only. Desktop reportedly exposes a single Replicate video model while the open praisonai-desktop engine has no video HTTP API.
This epic delivers provider registry + four cloud adapters + workflow tools + pricing catalog + desktop UI so agents, CLI, MCP, and desktop can execute the same routes users already priced in the picker.
Baseline architecture (as shipped)
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ YAML/Agent │ │ CLI videos │ │ MCP videos.gen │
└──────┬──────┘ └──────┬───────┘ └────────┬────────┘
│ │ │
└────────────────┼────────────────────┘
v
VideoAgent.generate()
v
litellm.videos.video_generation
v
openai | azure | gemini | vertex_ai | runwayml
Primary source files:
| Component | Path |
|---|---|
| VideoAgent | src/praisonai-agents/praisonaiagents/agent/video_agent.py |
| VideoConfig | same file — seconds, size, poll_interval, max_wait_time, timeout |
| Capabilities | src/praisonai/praisonai/capabilities/videos.py |
| CLI | src/praisonai-code/praisonai_code/cli/features/capabilities.py → handle_videos |
| MCP | src/praisonai-mcp/praisonai_mcp/mcp_server/adapters/extended_capabilities.py |
| Unit tests | src/praisonai-agents/tests/test_video_agent.py |
LiteLLM-supported llm= values (documented):
openai/sora-2,openai/sora-2-proazure/sora-2,azure/sora-2-progemini/veo-3.1-generate-preview,-fast-,-lite-vertex_ai/veo-3.1-generate-001,-fast-,-lite-runwayml/gen4_turbo
Target architecture
VideoAgent / video_generate / CLI / MCP / Desktop engine
│
v
VideoProviderRegistry.resolve(model)
│
┌──────────────────┼──────────────────┬─────────────┐
v v v v
LiteLLMAdapter OpenRouterProvider ReplicateProvider FalProvider
│ │ │ │
v v v v
Sora/Veo/Runway OR /videos API Predictions API fal queue
│
ViduProvider + DashScopeWanProvider
│
Workflow tools (ffmpeg / ComfyUI) — not providers
Companion matrix: paste SUPPORT-MATRIX.md into wiki for route-by-route status.
Child issues and technical scope
| # | Issue | Technical deliverable |
|---|---|---|
| 02 | Provider registry | VideoProvider protocol, namespaced job ids, unify 5 entry points |
| 03 | OpenRouter video | REST client: POST/GET /videos, download content; header parity with chat |
| 04 | Replicate video | Predictions lifecycle, per-model input profiles, MiniMax preset |
| 05 | fal video | Queue API, LTX 6s min, Omni two-frame |
| 06 | Vidu + DashScope | Direct APIs, off-peak queue, KF2V 5s fixed |
| 07 | ffmpeg / ComfyUI | Tools: ambient-loop, composite-freeze, LTX guide/API |
| 08 | Cost catalog | JSON schema, sync script, estimate_job, CLI compare |
| 09 | Desktop | /video/* engine routes + UI picker |
| 10 | Docs | Matrix, examples, CI guard, deprecation table |
Phased rollout
Phase 1 — Foundation (blocks everything)
- #02 registry + LiteLLM adapter extraction
- #10 matrix skeleton (all rows
planned) - #08 catalog static JSON (picker snapshot)
Phase 2 — Cloud channels (parallel)
- #03 OpenRouter (largest SKU coverage)
- #04 Replicate (desktop + cheap routes)
- #05 fal
- #06 direct vendors
Phase 3 — Product
- #07 workflows
- #09 desktop
- #08 estimator UI wiring
Epic acceptance criteria
- Coverage: ≥3 picker “cheap paid” routes + 3 workflow/free routes supported with mocked e2e tests
- Regression: Sora/Veo/Runway examples unchanged behavior
- Matrix: 100% picker routes documented with status
- Credentials:
.env.examplecomplete; onboarding mentions video keys - Desktop: Multi-model picker when keys configured
- Estimator: Default 8 clips × 5s scenario reproduces picker job costs within 1% on fixture catalog
Risks
| Risk | Mitigation |
|---|---|
| LiteLLM adds OpenRouter/Replicate video | Wrap in adapter; don’t fork vendor HTTP in two places |
| Replicate schema drift | Version-pinned profiles + catalog version bump |
| Long Vidu off-peak jobs | Async job docs; don’t use 600s default wait |
| TS/Rust parity expectations | #10 clarifies Python-only generation scope |
Out of scope
- Full Google Flow product clone
- HeyGen avatar pipeline (existing examples)
- TypeScript VideoAgent generation unless follow-up epic
References
- Cost picker: https://claude.ai/artifact/YaJ9iq9Tmich21NsE2jBc2
- OpenRouter video API: https://openrouter.ai/docs/api/api-reference/video-generation/submit-a-video-generation-request
- Examples:
examples/python/video/ - Support matrix document: SUPPORT-MATRIX.md
Appendix: Stakeholders and success metrics
- Agent developers can run picker routes via
VideoAgentwithout custom HTTP code. - Desktop users select models from catalog, not a single hardcoded Replicate slug.
- Finance/creative leads can estimate batch cost before generation (#08).
- Support team has single matrix (#10) for 'is route X supported?'
Appendix: Weekly milestone sketch
- Week 1–2: Registry (#02) + matrix skeleton (#10) + catalog JSON (#08).
- Week 3–4: OpenRouter (#03) + Replicate phase-1 (#04).
- Week 5: fal (#05) + DashScope/Vidu spike (#06).
- Week 6: ffmpeg workflows (#07) + desktop engine API (#09).
- Week 7: Hardening, docs, CI guards, beta desktop UI.
Appendix: Observability (epic-wide)
- Emit structured log fields:
provider,model,job_id,duration_ms,status. - Optional OpenTelemetry span:
video.generate→video.poll→video.download. - Counter metrics:
praisonai_video_jobs_total{provider,status}. - Histogram:
praisonai_video_job_duration_seconds{provider}.
Appendix: Security (epic-wide)
- Never log raw API keys, Authorization headers, or full prompts at INFO level.
- Redact secrets in desktop engine logs (see
engine/test_settings_redaction.py). - Validate user-supplied URLs in Replicate/fal download steps to prevent SSRF (allowlist replicate.delivery, fal.media domains).
- Cap upload size for
input_referenceimages (e.g. 20 MB) before provider upload. - Use HTTPS only for all provider base URLs; reject
http://overrides unless explicit dev flag.
Appendix: Glossary
- Route — Named row in cost picker (e.g. p-video draft).
- Channel — Billing path: OpenRouter, Replicate, fal, direct, LiteLLM-native.
- Provider adapter — Python class implementing VideoProvider protocol.
- Workflow — ffmpeg/ComfyUI tool; not a cloud VideoProvider.
- Catalog — JSON list of routes, pricing, capabilities (#08).
Appendix: Epic-level definition of done checklist
- All child issues closed or explicitly deferred with matrix status
not_planned. - CI green on mocked e2e for 3 cloud + 2 workflow routes.
- No regression in
tests/test_video_agent.pyfor LiteLLM mocks. - Published docs linked from README.
- Desktop beta can save MP4 from ≥3 providers when keys configured.
Cross-issue dependency matrix
| From | To | Contract |
|---|---|---|
| #02 | #03 | OpenRouterVideoProvider registered |
| #02 | #04 | ReplicateVideoProvider registered |
| #08 | #03 | Catalog sync from GET /videos/models |
| #09 | #02 | Engine calls VideoAgent only |
| #10 | all | Matrix status updated on merge |
Quality gates (CI)
pytest src/praisonai-agents/tests/video/pytest src/praisonai-desktop/engine/test_video_routes.py- Docs link checker for
docs/video/SUPPORT_MATRIX.md - Optional live smoke job gated by
PRAISONAI_LIVE_VIDEO=1
Release criteria
- All provider adapters merged with mocked HTTP tests green.
- SUPPORT_MATRIX.md lists every picker route with status.
- video_routes_catalog.json validates against JSON Schema in CI.
- Desktop engine exposes /video/generate behind existing auth gate.
- Examples exist for OpenRouter, Replicate, and fal minimum routes.
- Workflow tools ambient-loop and composite-freeze callable as @tool.
- No secrets in logs per engine redaction tests.
- CLI videos estimate subcommand returns JSON for 8×5 default scenario.
- MCP videos.generate accepts openrouter/ and replicate/ model prefixes.
- Documentation clarifies TS VideoAgent is analysis-only.
- Deprecation table published for Veo 3.0 and runway-gen3 MCP default.
- CHANGELOG entry summarizes breaking registry job id format.
- Performance: poll loop respects VideoConfig.max_wait_time.
- Security: SSRF allowlist enforced on provider download URLs.
- User-facing INSTALL documents required env vars per provider.
Picker route coverage targets
| Tier | Routes | Target count |
|---|---|---|
| Tier 0 workflow | ambient-loop, composite, LTX doc | 3 |
| Tier 1 cheap cloud | p-video draft, Wan 2.2 i2v, LTX fal | 3 |
| Tier 2 OpenRouter | Kling 3, Hailuo, Seedance | 3 |
| Tier 3 premium | Veo OR resold, Sora, Runway | 3 |
Epic communication checklist
- Post matrix link in #video channel when #10 skeleton merges.
- Weekly demo: one new picker route end-to-end per sprint.
- Track deferred routes with
not_planned+ reason in matrix. - Align desktop beta timeline with #09 engine API freeze.
- Coordinate breaking job id change with downstream recipe examples.
Additional verification notes
- Validate epic acceptance against SUPPORT_MATRIX row count equals picker route count.
- Confirm OpenRouter catalog sync job runs without OPENROUTER_API_KEY in CI (uses fixture).
- Confirm Replicate profiles pin version hashes in catalog JSON.
- Verify desktop engine never stores MP4 in logs-only temp without user consent path.
- Run regression: YAML workflow VideoAgent smoke with mocked LiteLLM adapter.
- Document Flow vs Veo API billing difference in BILLING_CAVEATS.md (#10).
- Ensure MCP tool schema documents seconds as string matching VideoConfig.
- Ensure CLI videos command accepts namespaced job id on status subcommand (future).
- Review child issue order in INDEX before filing on GitHub.
- Confirm LiteLLM adapter remains default when model has no recognized prefix.
- Add epic label and project board column for video parity tracking.
- Verify examples/python/video/README lists minimum env vars per provider.
- Check TS/Rust VideoAgent docs explicitly state analysis-only scope.
- Run support matrix diff in PR template for any new picker route.
- Schedule security review for SSRF allowlist before desktop beta.
Reference artifact: Video Model Cost Picker
Use this document as the epic body or project wiki appendix.
1. Runtime stack (Python generation)
1.1 VideoAgent call chain
| Step | Implementation |
|---|---|
| Construct | VideoAgent(llm=..., video=VideoConfig(seconds="8", size="1280x720", poll_interval=10, max_wait_time=600, timeout=600)) |
| Generate | generate(prompt, seconds=, size=, input_reference=, **kwargs) |
| Poll | wait_for_completion(video_id) — loops status() until completed or failed |
| Download | content(video_id) → bytes; or download() helper if present on agent |
Source: src/praisonai-agents/praisonaiagents/agent/video_agent.py
1.2 LiteLLM functions invoked
| Method | LiteLLM callable |
|---|---|
| Sync generate | litellm.videos.video_generation |
| Async generate | litellm.videos.avideo_generation |
| Status | video_status / avideo_status |
| Bytes | video_content / avideo_content |
| List | video_list / avideo_list |
| Remix | video_remix / avideo_remix |
1.3 LiteLLM video provider implementations (upstream)
| Prefix | Backend module pattern |
|---|---|
openai/ |
litellm/llms/openai/videos/transformation.py |
azure/ |
litellm/llms/azure/videos/transformation.py |
gemini/ |
litellm/llms/gemini/videos/transformation.py |
vertex_ai/ |
litellm/llms/vertex_ai/videos/transformation.py |
runwayml/ |
litellm/llms/runwayml/videos/transformation.py |
No upstream LiteLLM video handler for: replicate/, openrouter/, fal/, vidu/, dashscope/.
1.4 Parallel entry points (must stay consistent after registry #02)
| API | Module | Default model |
|---|---|---|
| SDK | VideoAgent |
openai/sora-2 |
| Capability | praisonai.capabilities.video_generate |
runway/gen3a_turbo |
| CLI | praisonai videos |
sora |
| MCP | praisonai.videos.generate |
runway-gen3 |
2. Supported model strings today (LiteLLM path)
llm= |
Env vars | Picker analog |
|---|---|---|
openai/sora-2 |
OPENAI_API_KEY |
sora_2 |
openai/sora-2-pro |
OPENAI_API_KEY |
sora_2_pro |
azure/sora-2 |
Azure OpenAI vars | Azure Sora |
gemini/veo-3.1-generate-preview |
GOOGLE_API_KEY / GEMINI_API_KEY |
Veo 3.1 full (partial channel) |
gemini/veo-3.1-fast-generate-preview |
same | Veo 3.1 Fast |
gemini/veo-3.1-lite-generate-preview |
same | Veo 3.1 Lite |
vertex_ai/veo-3.1-generate-001 |
GCP credentials | Veo 3.1 Vertex silent |
vertex_ai/veo-3.1-fast-generate-001 |
GCP | Veo 3.1 Fast |
vertex_ai/veo-3.1-lite-generate-001 |
GCP | Veo 3.1 Lite |
runwayml/gen4_turbo |
Runway API secret | Runway Gen-4 (not Gen-4.5 Replicate listing) |
Veo kwargs (LiteLLM passthrough): last_frame, reference_images for 3.1 interpolation.
3. Cost picker routes — full status table
Status codes: S Supported · P Partial (family only) · M Missing · W Workflow (offline tool)
| Route | Channel in picker | Status | PraisonAI target (llm= / tool) |
|---|---|---|---|
| ambient-loop | ffmpeg | W | tool:video_ambient_loop (#07) |
| Local composite | ffmpeg + mask | M→W | tool:video_composite_freeze_background (#07) |
| LTX Cinemagraph LoRA | ComfyUI | W | docs + optional ComfyUI API (#07) |
| MiniMax H3 Max free | fal | M | fal/minimax-h3-max (#05) |
| LTX-2.3 Fast | fal | M | fal/ltx-2.3-fast (#05) |
| p-video (draft) | Replicate | M | replicate/... + draft:true (#04) |
| p-video | Replicate | M | replicate/... (#04) |
| Wan 2.2 i2v Fast | Replicate | M | flat $0.05/run (#04) |
| Seedance 1.5 Pro | Replicate / OR | M | generate_audio:false (#04/#03) |
| Luma Ray 3.2 | Replicate | M | loop:true, not with end_image (#04) |
| Vidu Q3 Turbo | Vidu | M | vidu/q3-turbo (#06) |
| Wan 2.6 Flash | Alibaba | M | dashscope/wan2.6-flash (#06) |
| Kling 2.1 | Replicate | M | end_image + mode=pro (#04) |
| Wan 1.3B inpaint | Replicate | M | mask schema (#04) |
| Veo 3.1 Lite | Vertex / OR | P / M | S: gemini/vertex lite; M: OpenRouter resell (#03) |
| Wan 2.2 KF2V Flash | Alibaba | M | 5s fixed (#06) |
| MiniMax Hailuo 3 Max | OpenRouter | M | openrouter/... (#03) |
| Wan VACE 14B inpaint | Replicate | M | src_mask (#04) |
| Kling 3.0 Std / Pro | OpenRouter | M | (#03) |
| Wan 2.7 | OR / Alibaba | M | (#03/#06) |
| Runway Gen-4.5 | Runway / Replicate | P / M | P: runwayml/gen4_turbo; M: Replicate SKU (#04) |
| Veo 3.1 Fast | Vertex / OR | P / M | (#03 + gemini fast) |
| MiniMax Hailuo 3 | OpenRouter | M | (#03) |
| Wan VACE 14B | Replicate | M | (#04) |
| Gemini Omni Flash 1.1 | Gemini + fal | M | (#05 + future Gemini API) |
| Wan 3.0 | OpenRouter | M | keyframe warning (#03) |
| Veo 3.1 full | Vertex / OR | P / M | gemini/vertex standard (#03) |
| FLUX 3 Video | OpenRouter | M | (#03) |
| Seedance 2.0 | OpenRouter | M | token billing (#03) |
| MiniMax video-01 | Replicate | M | desktop preset (#04/#09) |
4. Non-generation video code (do not confuse with matrix)
| Package | Behavior |
|---|---|
praisonai-ts VideoAgent |
Multimodal analysis; default llm gpt-4o |
praisonai-rust VideoAgent |
Stub generate() |
praisonai_tools.video |
ffmpeg edit, transcribe, HTML motion graphics |
| HeyGen examples | Avatar + TTS pipeline |
OpenRouterProvider |
Chat completions only |
5. Catalog sync sources (for issue #08)
| Source | Endpoint / method |
|---|---|
| OpenRouter | GET https://openrouter.ai/api/v1/videos/models |
| Replicate | Per-model OpenAPI schema + pricing on model page |
| Picker manual rows | Alibaba direct $/s, Vidu off-peak, workflow $0 |
6. Summary metrics
| Metric | Value |
|---|---|
| LiteLLM-native families | 5 |
| Picker routes (approx.) | 30+ |
| Supported end-to-end today | 4 cloud families + 0 workflow tools packaged |
| Planned provider adapters | OpenRouter, Replicate, fal, Vidu, DashScope |
| Desktop video API in open engine | None (issue #09) |
- 主要言語
- Python
- スター
- 9.3k
- フォーク
- 1.5k
- 平均マージ
- 1日 5時間
- マージ済み PR(30日)
- 520
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
MervinPraison/PraisonAI のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
MervinPraison/PraisonAI#6710 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
claude
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
MervinPraison/PraisonAI#6707 ·
メンテナーはふだん 1 日以内に返信
-
bug claude
難易度 1/5 1時間未満 初心者へのやさしさ 90/100
MervinPraison/PraisonAI#6671 ·
メンテナーはふだん 1 日以内に返信
-
Compaction budget regression fixture remains below budget with accurate tokenization再び着手できるかも このイシューのプルリクエストはマージされずにクローズされました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
MervinPraison/PraisonAI#5418 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
-
bug claude documentation
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
MervinPraison/PraisonAI#5341 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
MervinPraison/PraisonAI の issue をすべて見る
似ている issue
-
cognition
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 78/100
collabnix/awesome-mcp-lists#433 ·
メンテナーはふだん 4 日以内に返信
-
CrossEntropyFilter raises on zero probabilities and single-class results対応中かも @NopeCharon が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
datajoint/datajoint-python#1703 ·
メンテナーはふだん 6 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
Archmonger/ServeStatic#129 ·