[epic][video] Cost Picker parity — multi-provider video generation
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 18/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Da chiarire
- Stato di attività
- Attiva
- Stack tecnologico
- python
- Ambito
- ai-infra-agents, api
Direzione di ricerca
This is an epic with no single change to make. Start by reading src/praisonai-agents/praisonaiagents/agent/video_agent.py and src/praisonai/praisonai/capabilities/videos.py to see how VideoAgent.generate() currently routes through LiteLLM. Done means one child deliverable (such as the provider registry, #02) is agreed with maintainers and merged with mocked HTTP tests; the epic itself is not a starter task.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Executive summary
The Video Model Cost Picker is a decision support tool for music-video and short-clip production: it compares 30+ routes spanning zero-cost ffmpeg/ComfyUI workflows, Replicate predictions, fal.ai queues, OpenRouter video jobs, direct Vidu/Alibaba APIs, and first-party OpenAI/Google/Runway APIs.
PraisonAI Python VideoAgent currently generates video only through LiteLLM, which implements five upstream families: OpenAI Sora, Azure Sora, Gemini Veo 3.1, Vertex Veo 3.1, RunwayML. Every other picker row is unsupported in code. OpenRouter integration today is chat-only (praisonai.llm.gateways.OpenRouterProvider). Replicate is LLM examples only. Desktop reportedly exposes a single Replicate video model while the open praisonai-desktop engine has no video HTTP API.
This epic delivers provider registry + four cloud adapters + workflow tools + pricing catalog + desktop UI so agents, CLI, MCP, and desktop can execute the same routes users already priced in the picker.
Baseline architecture (as shipped)
┌─────────────┐ ┌──────────────┐ ┌─────────────────┐
│ YAML/Agent │ │ CLI videos │ │ MCP videos.gen │
└──────┬──────┘ └──────┬───────┘ └────────┬────────┘
│ │ │
└────────────────┼────────────────────┘
v
VideoAgent.generate()
v
litellm.videos.video_generation
v
openai | azure | gemini | vertex_ai | runwayml
Primary source files:
| Component | Path |
|---|---|
| VideoAgent | src/praisonai-agents/praisonaiagents/agent/video_agent.py |
| VideoConfig | same file — seconds, size, poll_interval, max_wait_time, timeout |
| Capabilities | src/praisonai/praisonai/capabilities/videos.py |
| CLI | src/praisonai-code/praisonai_code/cli/features/capabilities.py → handle_videos |
| MCP | src/praisonai-mcp/praisonai_mcp/mcp_server/adapters/extended_capabilities.py |
| Unit tests | src/praisonai-agents/tests/test_video_agent.py |
LiteLLM-supported llm= values (documented):
openai/sora-2,openai/sora-2-proazure/sora-2,azure/sora-2-progemini/veo-3.1-generate-preview,-fast-,-lite-vertex_ai/veo-3.1-generate-001,-fast-,-lite-runwayml/gen4_turbo
Target architecture
VideoAgent / video_generate / CLI / MCP / Desktop engine
│
v
VideoProviderRegistry.resolve(model)
│
┌──────────────────┼──────────────────┬─────────────┐
v v v v
LiteLLMAdapter OpenRouterProvider ReplicateProvider FalProvider
│ │ │ │
v v v v
Sora/Veo/Runway OR /videos API Predictions API fal queue
│
ViduProvider + DashScopeWanProvider
│
Workflow tools (ffmpeg / ComfyUI) — not providers
Companion matrix: paste SUPPORT-MATRIX.md into wiki for route-by-route status.
Child issues and technical scope
| # | Issue | Technical deliverable |
|---|---|---|
| 02 | Provider registry | VideoProvider protocol, namespaced job ids, unify 5 entry points |
| 03 | OpenRouter video | REST client: POST/GET /videos, download content; header parity with chat |
| 04 | Replicate video | Predictions lifecycle, per-model input profiles, MiniMax preset |
| 05 | fal video | Queue API, LTX 6s min, Omni two-frame |
| 06 | Vidu + DashScope | Direct APIs, off-peak queue, KF2V 5s fixed |
| 07 | ffmpeg / ComfyUI | Tools: ambient-loop, composite-freeze, LTX guide/API |
| 08 | Cost catalog | JSON schema, sync script, estimate_job, CLI compare |
| 09 | Desktop | /video/* engine routes + UI picker |
| 10 | Docs | Matrix, examples, CI guard, deprecation table |
Phased rollout
Phase 1 — Foundation (blocks everything)
- #02 registry + LiteLLM adapter extraction
- #10 matrix skeleton (all rows
planned) - #08 catalog static JSON (picker snapshot)
Phase 2 — Cloud channels (parallel)
- #03 OpenRouter (largest SKU coverage)
- #04 Replicate (desktop + cheap routes)
- #05 fal
- #06 direct vendors
Phase 3 — Product
- #07 workflows
- #09 desktop
- #08 estimator UI wiring
Epic acceptance criteria
- Coverage: ≥3 picker “cheap paid” routes + 3 workflow/free routes supported with mocked e2e tests
- Regression: Sora/Veo/Runway examples unchanged behavior
- Matrix: 100% picker routes documented with status
- Credentials:
.env.examplecomplete; onboarding mentions video keys - Desktop: Multi-model picker when keys configured
- Estimator: Default 8 clips × 5s scenario reproduces picker job costs within 1% on fixture catalog
Risks
| Risk | Mitigation |
|---|---|
| LiteLLM adds OpenRouter/Replicate video | Wrap in adapter; don’t fork vendor HTTP in two places |
| Replicate schema drift | Version-pinned profiles + catalog version bump |
| Long Vidu off-peak jobs | Async job docs; don’t use 600s default wait |
| TS/Rust parity expectations | #10 clarifies Python-only generation scope |
Out of scope
- Full Google Flow product clone
- HeyGen avatar pipeline (existing examples)
- TypeScript VideoAgent generation unless follow-up epic
References
- Cost picker: https://claude.ai/artifact/YaJ9iq9Tmich21NsE2jBc2
- OpenRouter video API: https://openrouter.ai/docs/api/api-reference/video-generation/submit-a-video-generation-request
- Examples:
examples/python/video/ - Support matrix document: SUPPORT-MATRIX.md
Appendix: Stakeholders and success metrics
- Agent developers can run picker routes via
VideoAgentwithout custom HTTP code. - Desktop users select models from catalog, not a single hardcoded Replicate slug.
- Finance/creative leads can estimate batch cost before generation (#08).
- Support team has single matrix (#10) for 'is route X supported?'
Appendix: Weekly milestone sketch
- Week 1–2: Registry (#02) + matrix skeleton (#10) + catalog JSON (#08).
- Week 3–4: OpenRouter (#03) + Replicate phase-1 (#04).
- Week 5: fal (#05) + DashScope/Vidu spike (#06).
- Week 6: ffmpeg workflows (#07) + desktop engine API (#09).
- Week 7: Hardening, docs, CI guards, beta desktop UI.
Appendix: Observability (epic-wide)
- Emit structured log fields:
provider,model,job_id,duration_ms,status. - Optional OpenTelemetry span:
video.generate→video.poll→video.download. - Counter metrics:
praisonai_video_jobs_total{provider,status}. - Histogram:
praisonai_video_job_duration_seconds{provider}.
Appendix: Security (epic-wide)
- Never log raw API keys, Authorization headers, or full prompts at INFO level.
- Redact secrets in desktop engine logs (see
engine/test_settings_redaction.py). - Validate user-supplied URLs in Replicate/fal download steps to prevent SSRF (allowlist replicate.delivery, fal.media domains).
- Cap upload size for
input_referenceimages (e.g. 20 MB) before provider upload. - Use HTTPS only for all provider base URLs; reject
http://overrides unless explicit dev flag.
Appendix: Glossary
- Route — Named row in cost picker (e.g. p-video draft).
- Channel — Billing path: OpenRouter, Replicate, fal, direct, LiteLLM-native.
- Provider adapter — Python class implementing VideoProvider protocol.
- Workflow — ffmpeg/ComfyUI tool; not a cloud VideoProvider.
- Catalog — JSON list of routes, pricing, capabilities (#08).
Appendix: Epic-level definition of done checklist
- All child issues closed or explicitly deferred with matrix status
not_planned. - CI green on mocked e2e for 3 cloud + 2 workflow routes.
- No regression in
tests/test_video_agent.pyfor LiteLLM mocks. - Published docs linked from README.
- Desktop beta can save MP4 from ≥3 providers when keys configured.
Cross-issue dependency matrix
| From | To | Contract |
|---|---|---|
| #02 | #03 | OpenRouterVideoProvider registered |
| #02 | #04 | ReplicateVideoProvider registered |
| #08 | #03 | Catalog sync from GET /videos/models |
| #09 | #02 | Engine calls VideoAgent only |
| #10 | all | Matrix status updated on merge |
Quality gates (CI)
pytest src/praisonai-agents/tests/video/pytest src/praisonai-desktop/engine/test_video_routes.py- Docs link checker for
docs/video/SUPPORT_MATRIX.md - Optional live smoke job gated by
PRAISONAI_LIVE_VIDEO=1
Release criteria
- All provider adapters merged with mocked HTTP tests green.
- SUPPORT_MATRIX.md lists every picker route with status.
- video_routes_catalog.json validates against JSON Schema in CI.
- Desktop engine exposes /video/generate behind existing auth gate.
- Examples exist for OpenRouter, Replicate, and fal minimum routes.
- Workflow tools ambient-loop and composite-freeze callable as @tool.
- No secrets in logs per engine redaction tests.
- CLI videos estimate subcommand returns JSON for 8×5 default scenario.
- MCP videos.generate accepts openrouter/ and replicate/ model prefixes.
- Documentation clarifies TS VideoAgent is analysis-only.
- Deprecation table published for Veo 3.0 and runway-gen3 MCP default.
- CHANGELOG entry summarizes breaking registry job id format.
- Performance: poll loop respects VideoConfig.max_wait_time.
- Security: SSRF allowlist enforced on provider download URLs.
- User-facing INSTALL documents required env vars per provider.
Picker route coverage targets
| Tier | Routes | Target count |
|---|---|---|
| Tier 0 workflow | ambient-loop, composite, LTX doc | 3 |
| Tier 1 cheap cloud | p-video draft, Wan 2.2 i2v, LTX fal | 3 |
| Tier 2 OpenRouter | Kling 3, Hailuo, Seedance | 3 |
| Tier 3 premium | Veo OR resold, Sora, Runway | 3 |
Epic communication checklist
- Post matrix link in #video channel when #10 skeleton merges.
- Weekly demo: one new picker route end-to-end per sprint.
- Track deferred routes with
not_planned+ reason in matrix. - Align desktop beta timeline with #09 engine API freeze.
- Coordinate breaking job id change with downstream recipe examples.
Additional verification notes
- Validate epic acceptance against SUPPORT_MATRIX row count equals picker route count.
- Confirm OpenRouter catalog sync job runs without OPENROUTER_API_KEY in CI (uses fixture).
- Confirm Replicate profiles pin version hashes in catalog JSON.
- Verify desktop engine never stores MP4 in logs-only temp without user consent path.
- Run regression: YAML workflow VideoAgent smoke with mocked LiteLLM adapter.
- Document Flow vs Veo API billing difference in BILLING_CAVEATS.md (#10).
- Ensure MCP tool schema documents seconds as string matching VideoConfig.
- Ensure CLI videos command accepts namespaced job id on status subcommand (future).
- Review child issue order in INDEX before filing on GitHub.
- Confirm LiteLLM adapter remains default when model has no recognized prefix.
- Add epic label and project board column for video parity tracking.
- Verify examples/python/video/README lists minimum env vars per provider.
- Check TS/Rust VideoAgent docs explicitly state analysis-only scope.
- Run support matrix diff in PR template for any new picker route.
- Schedule security review for SSRF allowlist before desktop beta.
Reference artifact: Video Model Cost Picker
Use this document as the epic body or project wiki appendix.
1. Runtime stack (Python generation)
1.1 VideoAgent call chain
| Step | Implementation |
|---|---|
| Construct | VideoAgent(llm=..., video=VideoConfig(seconds="8", size="1280x720", poll_interval=10, max_wait_time=600, timeout=600)) |
| Generate | generate(prompt, seconds=, size=, input_reference=, **kwargs) |
| Poll | wait_for_completion(video_id) — loops status() until completed or failed |
| Download | content(video_id) → bytes; or download() helper if present on agent |
Source: src/praisonai-agents/praisonaiagents/agent/video_agent.py
1.2 LiteLLM functions invoked
| Method | LiteLLM callable |
|---|---|
| Sync generate | litellm.videos.video_generation |
| Async generate | litellm.videos.avideo_generation |
| Status | video_status / avideo_status |
| Bytes | video_content / avideo_content |
| List | video_list / avideo_list |
| Remix | video_remix / avideo_remix |
1.3 LiteLLM video provider implementations (upstream)
| Prefix | Backend module pattern |
|---|---|
openai/ |
litellm/llms/openai/videos/transformation.py |
azure/ |
litellm/llms/azure/videos/transformation.py |
gemini/ |
litellm/llms/gemini/videos/transformation.py |
vertex_ai/ |
litellm/llms/vertex_ai/videos/transformation.py |
runwayml/ |
litellm/llms/runwayml/videos/transformation.py |
No upstream LiteLLM video handler for: replicate/, openrouter/, fal/, vidu/, dashscope/.
1.4 Parallel entry points (must stay consistent after registry #02)
| API | Module | Default model |
|---|---|---|
| SDK | VideoAgent |
openai/sora-2 |
| Capability | praisonai.capabilities.video_generate |
runway/gen3a_turbo |
| CLI | praisonai videos |
sora |
| MCP | praisonai.videos.generate |
runway-gen3 |
2. Supported model strings today (LiteLLM path)
llm= |
Env vars | Picker analog |
|---|---|---|
openai/sora-2 |
OPENAI_API_KEY |
sora_2 |
openai/sora-2-pro |
OPENAI_API_KEY |
sora_2_pro |
azure/sora-2 |
Azure OpenAI vars | Azure Sora |
gemini/veo-3.1-generate-preview |
GOOGLE_API_KEY / GEMINI_API_KEY |
Veo 3.1 full (partial channel) |
gemini/veo-3.1-fast-generate-preview |
same | Veo 3.1 Fast |
gemini/veo-3.1-lite-generate-preview |
same | Veo 3.1 Lite |
vertex_ai/veo-3.1-generate-001 |
GCP credentials | Veo 3.1 Vertex silent |
vertex_ai/veo-3.1-fast-generate-001 |
GCP | Veo 3.1 Fast |
vertex_ai/veo-3.1-lite-generate-001 |
GCP | Veo 3.1 Lite |
runwayml/gen4_turbo |
Runway API secret | Runway Gen-4 (not Gen-4.5 Replicate listing) |
Veo kwargs (LiteLLM passthrough): last_frame, reference_images for 3.1 interpolation.
3. Cost picker routes — full status table
Status codes: S Supported · P Partial (family only) · M Missing · W Workflow (offline tool)
| Route | Channel in picker | Status | PraisonAI target (llm= / tool) |
|---|---|---|---|
| ambient-loop | ffmpeg | W | tool:video_ambient_loop (#07) |
| Local composite | ffmpeg + mask | M→W | tool:video_composite_freeze_background (#07) |
| LTX Cinemagraph LoRA | ComfyUI | W | docs + optional ComfyUI API (#07) |
| MiniMax H3 Max free | fal | M | fal/minimax-h3-max (#05) |
| LTX-2.3 Fast | fal | M | fal/ltx-2.3-fast (#05) |
| p-video (draft) | Replicate | M | replicate/... + draft:true (#04) |
| p-video | Replicate | M | replicate/... (#04) |
| Wan 2.2 i2v Fast | Replicate | M | flat $0.05/run (#04) |
| Seedance 1.5 Pro | Replicate / OR | M | generate_audio:false (#04/#03) |
| Luma Ray 3.2 | Replicate | M | loop:true, not with end_image (#04) |
| Vidu Q3 Turbo | Vidu | M | vidu/q3-turbo (#06) |
| Wan 2.6 Flash | Alibaba | M | dashscope/wan2.6-flash (#06) |
| Kling 2.1 | Replicate | M | end_image + mode=pro (#04) |
| Wan 1.3B inpaint | Replicate | M | mask schema (#04) |
| Veo 3.1 Lite | Vertex / OR | P / M | S: gemini/vertex lite; M: OpenRouter resell (#03) |
| Wan 2.2 KF2V Flash | Alibaba | M | 5s fixed (#06) |
| MiniMax Hailuo 3 Max | OpenRouter | M | openrouter/... (#03) |
| Wan VACE 14B inpaint | Replicate | M | src_mask (#04) |
| Kling 3.0 Std / Pro | OpenRouter | M | (#03) |
| Wan 2.7 | OR / Alibaba | M | (#03/#06) |
| Runway Gen-4.5 | Runway / Replicate | P / M | P: runwayml/gen4_turbo; M: Replicate SKU (#04) |
| Veo 3.1 Fast | Vertex / OR | P / M | (#03 + gemini fast) |
| MiniMax Hailuo 3 | OpenRouter | M | (#03) |
| Wan VACE 14B | Replicate | M | (#04) |
| Gemini Omni Flash 1.1 | Gemini + fal | M | (#05 + future Gemini API) |
| Wan 3.0 | OpenRouter | M | keyframe warning (#03) |
| Veo 3.1 full | Vertex / OR | P / M | gemini/vertex standard (#03) |
| FLUX 3 Video | OpenRouter | M | (#03) |
| Seedance 2.0 | OpenRouter | M | token billing (#03) |
| MiniMax video-01 | Replicate | M | desktop preset (#04/#09) |
4. Non-generation video code (do not confuse with matrix)
| Package | Behavior |
|---|---|
praisonai-ts VideoAgent |
Multimodal analysis; default llm gpt-4o |
praisonai-rust VideoAgent |
Stub generate() |
praisonai_tools.video |
ffmpeg edit, transcribe, HTML motion graphics |
| HeyGen examples | Avatar + TTS pipeline |
OpenRouterProvider |
Chat completions only |
5. Catalog sync sources (for issue #08)
| Source | Endpoint / method |
|---|---|
| OpenRouter | GET https://openrouter.ai/api/v1/videos/models |
| Replicate | Per-model OpenAPI schema + pricing on model page |
| Picker manual rows | Alibaba direct $/s, Vidu off-peak, workflow $0 |
6. Summary metrics
| Metric | Value |
|---|---|
| LiteLLM-native families | 5 |
| Picker routes (approx.) | 30+ |
| Supported end-to-end today | 4 cloud families + 0 workflow tools packaged |
| Planned provider adapters | OpenRouter, Replicate, fal, Vidu, DashScope |
| Desktop video API in open engine | None (issue #09) |
- Lingua principale
- Python
- Stelle
- 9.3k
- Fork
- 1.5k
- Merge medio
- 1g 5h
- PR unite (30g)
- 520
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di MervinPraison/PraisonAI
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
MervinPraison/PraisonAI#6710 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
claude
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
MervinPraison/PraisonAI#6707 ·
I maintainer di solito rispondono entro 1 giorno
-
bug claude
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 90/100
MervinPraison/PraisonAI#6671 ·
I maintainer di solito rispondono entro 1 giorno
-
Compaction budget regression fixture remains below budget with accurate tokenizationForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
MervinPraison/PraisonAI#5418 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
bug claude documentation
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
MervinPraison/PraisonAI#5341 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di MervinPraison/PraisonAI
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
I maintainer di solito rispondono entro 2 giorni
-
[Bug]: LANGSMITH_SAMPLING_RATE ignores fractional values (isdigit check), traces 100% of requestsAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
I maintainer di solito rispondono entro 1 giorno
-
platform/backend tests
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
Significant-Gravitas/AutoGPT#17100 ·
I maintainer di solito rispondono entro 1 giorno
-
bug good first issue
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 61/100
scverse/infercnvpy#238 ·
I maintainer di solito rispondono entro 1 giorno