[epic][video] Cost Picker parity — multi-provider video generation

オープン
#5,171 コメント 4 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
18/100
issue の種類
機能追加
明瞭さ
説明が足りない
活発さ
活発
技術スタック
python

調査の方向性

This is an epic with no single change to make. Start by reading src/praisonai-agents/praisonaiagents/agent/video_agent.py and src/praisonai/praisonai/capabilities/videos.py to see how VideoAgent.generate() currently routes through LiteLLM. Done means one child deliverable (such as the provider registry, #02) is agreed with maintainers and merged with mocked HTTP tests; the epic itself is not a starter task.

索引モデルが issue の本文から書いたものです。

説明

documentation javascript origin/outsider performance security

Executive summary

The Video Model Cost Picker is a decision support tool for music-video and short-clip production: it compares 30+ routes spanning zero-cost ffmpeg/ComfyUI workflows, Replicate predictions, fal.ai queues, OpenRouter video jobs, direct Vidu/Alibaba APIs, and first-party OpenAI/Google/Runway APIs.

PraisonAI Python VideoAgent currently generates video only through LiteLLM, which implements five upstream families: OpenAI Sora, Azure Sora, Gemini Veo 3.1, Vertex Veo 3.1, RunwayML. Every other picker row is unsupported in code. OpenRouter integration today is chat-only (praisonai.llm.gateways.OpenRouterProvider). Replicate is LLM examples only. Desktop reportedly exposes a single Replicate video model while the open praisonai-desktop engine has no video HTTP API.

This epic delivers provider registry + four cloud adapters + workflow tools + pricing catalog + desktop UI so agents, CLI, MCP, and desktop can execute the same routes users already priced in the picker.


Baseline architecture (as shipped)

┌─────────────┐  ┌──────────────┐  ┌─────────────────┐
│ YAML/Agent  │  │ CLI videos   │  │ MCP videos.gen  │
└──────┬──────┘  └──────┬───────┘  └────────┬────────┘
       │                │                    │
       └────────────────┼────────────────────┘
                        v
              VideoAgent.generate()
                        v
              litellm.videos.video_generation
                        v
     openai | azure | gemini | vertex_ai | runwayml

Primary source files:

Component Path
VideoAgent src/praisonai-agents/praisonaiagents/agent/video_agent.py
VideoConfig same file — seconds, size, poll_interval, max_wait_time, timeout
Capabilities src/praisonai/praisonai/capabilities/videos.py
CLI src/praisonai-code/praisonai_code/cli/features/capabilities.py → handle_videos
MCP src/praisonai-mcp/praisonai_mcp/mcp_server/adapters/extended_capabilities.py
Unit tests src/praisonai-agents/tests/test_video_agent.py

LiteLLM-supported llm= values (documented):

  • openai/sora-2, openai/sora-2-pro
  • azure/sora-2, azure/sora-2-pro
  • gemini/veo-3.1-generate-preview, -fast-, -lite-
  • vertex_ai/veo-3.1-generate-001, -fast-, -lite-
  • runwayml/gen4_turbo

Target architecture

VideoAgent / video_generate / CLI / MCP / Desktop engine
                        │
                        v
           VideoProviderRegistry.resolve(model)
                        │
     ┌──────────────────┼──────────────────┬─────────────┐
     v                  v                  v             v
LiteLLMAdapter   OpenRouterProvider  ReplicateProvider  FalProvider
     │                  │                  │             │
     v                  v                  v             v
  Sora/Veo/Runway   OR /videos API    Predictions API   fal queue
                        │
                   ViduProvider + DashScopeWanProvider
                        │
              Workflow tools (ffmpeg / ComfyUI) — not providers

Companion matrix: paste SUPPORT-MATRIX.md into wiki for route-by-route status.


Child issues and technical scope

# Issue Technical deliverable
02 Provider registry VideoProvider protocol, namespaced job ids, unify 5 entry points
03 OpenRouter video REST client: POST/GET /videos, download content; header parity with chat
04 Replicate video Predictions lifecycle, per-model input profiles, MiniMax preset
05 fal video Queue API, LTX 6s min, Omni two-frame
06 Vidu + DashScope Direct APIs, off-peak queue, KF2V 5s fixed
07 ffmpeg / ComfyUI Tools: ambient-loop, composite-freeze, LTX guide/API
08 Cost catalog JSON schema, sync script, estimate_job, CLI compare
09 Desktop /video/* engine routes + UI picker
10 Docs Matrix, examples, CI guard, deprecation table

Phased rollout

Phase 1 — Foundation (blocks everything)
  • #02 registry + LiteLLM adapter extraction
  • #10 matrix skeleton (all rows planned)
  • #08 catalog static JSON (picker snapshot)
Phase 2 — Cloud channels (parallel)
  • #03 OpenRouter (largest SKU coverage)
  • #04 Replicate (desktop + cheap routes)
  • #05 fal
  • #06 direct vendors
Phase 3 — Product
  • #07 workflows
  • #09 desktop
  • #08 estimator UI wiring

Epic acceptance criteria

  • Coverage: ≥3 picker “cheap paid” routes + 3 workflow/free routes supported with mocked e2e tests
  • Regression: Sora/Veo/Runway examples unchanged behavior
  • Matrix: 100% picker routes documented with status
  • Credentials: .env.example complete; onboarding mentions video keys
  • Desktop: Multi-model picker when keys configured
  • Estimator: Default 8 clips × 5s scenario reproduces picker job costs within 1% on fixture catalog

Risks

Risk Mitigation
LiteLLM adds OpenRouter/Replicate video Wrap in adapter; don’t fork vendor HTTP in two places
Replicate schema drift Version-pinned profiles + catalog version bump
Long Vidu off-peak jobs Async job docs; don’t use 600s default wait
TS/Rust parity expectations #10 clarifies Python-only generation scope

Out of scope

  • Full Google Flow product clone
  • HeyGen avatar pipeline (existing examples)
  • TypeScript VideoAgent generation unless follow-up epic

References


Appendix: Stakeholders and success metrics

  • Agent developers can run picker routes via VideoAgent without custom HTTP code.
  • Desktop users select models from catalog, not a single hardcoded Replicate slug.
  • Finance/creative leads can estimate batch cost before generation (#08).
  • Support team has single matrix (#10) for 'is route X supported?'

Appendix: Weekly milestone sketch

  1. Week 1–2: Registry (#02) + matrix skeleton (#10) + catalog JSON (#08).
  2. Week 3–4: OpenRouter (#03) + Replicate phase-1 (#04).
  3. Week 5: fal (#05) + DashScope/Vidu spike (#06).
  4. Week 6: ffmpeg workflows (#07) + desktop engine API (#09).
  5. Week 7: Hardening, docs, CI guards, beta desktop UI.

Appendix: Observability (epic-wide)

  • Emit structured log fields: provider, model, job_id, duration_ms, status.
  • Optional OpenTelemetry span: video.generate → video.poll → video.download.
  • Counter metrics: praisonai_video_jobs_total{provider,status}.
  • Histogram: praisonai_video_job_duration_seconds{provider}.

Appendix: Security (epic-wide)

  • Never log raw API keys, Authorization headers, or full prompts at INFO level.
  • Redact secrets in desktop engine logs (see engine/test_settings_redaction.py).
  • Validate user-supplied URLs in Replicate/fal download steps to prevent SSRF (allowlist replicate.delivery, fal.media domains).
  • Cap upload size for input_reference images (e.g. 20 MB) before provider upload.
  • Use HTTPS only for all provider base URLs; reject http:// overrides unless explicit dev flag.

Appendix: Glossary

  • Route — Named row in cost picker (e.g. p-video draft).
  • Channel — Billing path: OpenRouter, Replicate, fal, direct, LiteLLM-native.
  • Provider adapter — Python class implementing VideoProvider protocol.
  • Workflow — ffmpeg/ComfyUI tool; not a cloud VideoProvider.
  • Catalog — JSON list of routes, pricing, capabilities (#08).

Appendix: Epic-level definition of done checklist

  1. All child issues closed or explicitly deferred with matrix status not_planned.
  2. CI green on mocked e2e for 3 cloud + 2 workflow routes.
  3. No regression in tests/test_video_agent.py for LiteLLM mocks.
  4. Published docs linked from README.
  5. Desktop beta can save MP4 from ≥3 providers when keys configured.







Cross-issue dependency matrix

From To Contract
#02 #03 OpenRouterVideoProvider registered
#02 #04 ReplicateVideoProvider registered
#08 #03 Catalog sync from GET /videos/models
#09 #02 Engine calls VideoAgent only
#10 all Matrix status updated on merge

Quality gates (CI)

  • pytest src/praisonai-agents/tests/video/
  • pytest src/praisonai-desktop/engine/test_video_routes.py
  • Docs link checker for docs/video/SUPPORT_MATRIX.md
  • Optional live smoke job gated by PRAISONAI_LIVE_VIDEO=1

Release criteria

  • All provider adapters merged with mocked HTTP tests green.
  • SUPPORT_MATRIX.md lists every picker route with status.
  • video_routes_catalog.json validates against JSON Schema in CI.
  • Desktop engine exposes /video/generate behind existing auth gate.
  • Examples exist for OpenRouter, Replicate, and fal minimum routes.
  • Workflow tools ambient-loop and composite-freeze callable as @tool.
  • No secrets in logs per engine redaction tests.
  • CLI videos estimate subcommand returns JSON for 8×5 default scenario.
  • MCP videos.generate accepts openrouter/ and replicate/ model prefixes.
  • Documentation clarifies TS VideoAgent is analysis-only.
  • Deprecation table published for Veo 3.0 and runway-gen3 MCP default.
  • CHANGELOG entry summarizes breaking registry job id format.
  • Performance: poll loop respects VideoConfig.max_wait_time.
  • Security: SSRF allowlist enforced on provider download URLs.
  • User-facing INSTALL documents required env vars per provider.

Picker route coverage targets

Tier Routes Target count
Tier 0 workflow ambient-loop, composite, LTX doc 3
Tier 1 cheap cloud p-video draft, Wan 2.2 i2v, LTX fal 3
Tier 2 OpenRouter Kling 3, Hailuo, Seedance 3
Tier 3 premium Veo OR resold, Sora, Runway 3

Epic communication checklist

  1. Post matrix link in #video channel when #10 skeleton merges.
  2. Weekly demo: one new picker route end-to-end per sprint.
  3. Track deferred routes with not_planned + reason in matrix.
  4. Align desktop beta timeline with #09 engine API freeze.
  5. Coordinate breaking job id change with downstream recipe examples.

Additional verification notes

  • Validate epic acceptance against SUPPORT_MATRIX row count equals picker route count.
  • Confirm OpenRouter catalog sync job runs without OPENROUTER_API_KEY in CI (uses fixture).
  • Confirm Replicate profiles pin version hashes in catalog JSON.
  • Verify desktop engine never stores MP4 in logs-only temp without user consent path.
  • Run regression: YAML workflow VideoAgent smoke with mocked LiteLLM adapter.
  • Document Flow vs Veo API billing difference in BILLING_CAVEATS.md (#10).
  • Ensure MCP tool schema documents seconds as string matching VideoConfig.
  • Ensure CLI videos command accepts namespaced job id on status subcommand (future).
  • Review child issue order in INDEX before filing on GitHub.
  • Confirm LiteLLM adapter remains default when model has no recognized prefix.
  • Add epic label and project board column for video parity tracking.
  • Verify examples/python/video/README lists minimum env vars per provider.
  • Check TS/Rust VideoAgent docs explicitly state analysis-only scope.
  • Run support matrix diff in PR template for any new picker route.
  • Schedule security review for SSRF allowlist before desktop beta.

Reference artifact: Video Model Cost Picker

Use this document as the epic body or project wiki appendix.


1. Runtime stack (Python generation)

1.1 VideoAgent call chain
Step Implementation
Construct VideoAgent(llm=..., video=VideoConfig(seconds="8", size="1280x720", poll_interval=10, max_wait_time=600, timeout=600))
Generate generate(prompt, seconds=, size=, input_reference=, **kwargs)
Poll wait_for_completion(video_id) — loops status() until completed or failed
Download content(video_id) → bytes; or download() helper if present on agent

Source: src/praisonai-agents/praisonaiagents/agent/video_agent.py

1.2 LiteLLM functions invoked
Method LiteLLM callable
Sync generate litellm.videos.video_generation
Async generate litellm.videos.avideo_generation
Status video_status / avideo_status
Bytes video_content / avideo_content
List video_list / avideo_list
Remix video_remix / avideo_remix
1.3 LiteLLM video provider implementations (upstream)
Prefix Backend module pattern
openai/ litellm/llms/openai/videos/transformation.py
azure/ litellm/llms/azure/videos/transformation.py
gemini/ litellm/llms/gemini/videos/transformation.py
vertex_ai/ litellm/llms/vertex_ai/videos/transformation.py
runwayml/ litellm/llms/runwayml/videos/transformation.py

No upstream LiteLLM video handler for: replicate/, openrouter/, fal/, vidu/, dashscope/.

1.4 Parallel entry points (must stay consistent after registry #02)
API Module Default model
SDK VideoAgent openai/sora-2
Capability praisonai.capabilities.video_generate runway/gen3a_turbo
CLI praisonai videos sora
MCP praisonai.videos.generate runway-gen3

2. Supported model strings today (LiteLLM path)

llm= Env vars Picker analog
openai/sora-2 OPENAI_API_KEY sora_2
openai/sora-2-pro OPENAI_API_KEY sora_2_pro
azure/sora-2 Azure OpenAI vars Azure Sora
gemini/veo-3.1-generate-preview GOOGLE_API_KEY / GEMINI_API_KEY Veo 3.1 full (partial channel)
gemini/veo-3.1-fast-generate-preview same Veo 3.1 Fast
gemini/veo-3.1-lite-generate-preview same Veo 3.1 Lite
vertex_ai/veo-3.1-generate-001 GCP credentials Veo 3.1 Vertex silent
vertex_ai/veo-3.1-fast-generate-001 GCP Veo 3.1 Fast
vertex_ai/veo-3.1-lite-generate-001 GCP Veo 3.1 Lite
runwayml/gen4_turbo Runway API secret Runway Gen-4 (not Gen-4.5 Replicate listing)

Veo kwargs (LiteLLM passthrough): last_frame, reference_images for 3.1 interpolation.


3. Cost picker routes — full status table

Status codes: S Supported · P Partial (family only) · M Missing · W Workflow (offline tool)

Route Channel in picker Status PraisonAI target (llm= / tool)
ambient-loop ffmpeg W tool:video_ambient_loop (#07)
Local composite ffmpeg + mask M→W tool:video_composite_freeze_background (#07)
LTX Cinemagraph LoRA ComfyUI W docs + optional ComfyUI API (#07)
MiniMax H3 Max free fal M fal/minimax-h3-max (#05)
LTX-2.3 Fast fal M fal/ltx-2.3-fast (#05)
p-video (draft) Replicate M replicate/... + draft:true (#04)
p-video Replicate M replicate/... (#04)
Wan 2.2 i2v Fast Replicate M flat $0.05/run (#04)
Seedance 1.5 Pro Replicate / OR M generate_audio:false (#04/#03)
Luma Ray 3.2 Replicate M loop:true, not with end_image (#04)
Vidu Q3 Turbo Vidu M vidu/q3-turbo (#06)
Wan 2.6 Flash Alibaba M dashscope/wan2.6-flash (#06)
Kling 2.1 Replicate M end_image + mode=pro (#04)
Wan 1.3B inpaint Replicate M mask schema (#04)
Veo 3.1 Lite Vertex / OR P / M S: gemini/vertex lite; M: OpenRouter resell (#03)
Wan 2.2 KF2V Flash Alibaba M 5s fixed (#06)
MiniMax Hailuo 3 Max OpenRouter M openrouter/... (#03)
Wan VACE 14B inpaint Replicate M src_mask (#04)
Kling 3.0 Std / Pro OpenRouter M (#03)
Wan 2.7 OR / Alibaba M (#03/#06)
Runway Gen-4.5 Runway / Replicate P / M P: runwayml/gen4_turbo; M: Replicate SKU (#04)
Veo 3.1 Fast Vertex / OR P / M (#03 + gemini fast)
MiniMax Hailuo 3 OpenRouter M (#03)
Wan VACE 14B Replicate M (#04)
Gemini Omni Flash 1.1 Gemini + fal M (#05 + future Gemini API)
Wan 3.0 OpenRouter M keyframe warning (#03)
Veo 3.1 full Vertex / OR P / M gemini/vertex standard (#03)
FLUX 3 Video OpenRouter M (#03)
Seedance 2.0 OpenRouter M token billing (#03)
MiniMax video-01 Replicate M desktop preset (#04/#09)

4. Non-generation video code (do not confuse with matrix)

Package Behavior
praisonai-ts VideoAgent Multimodal analysis; default llm gpt-4o
praisonai-rust VideoAgent Stub generate()
praisonai_tools.video ffmpeg edit, transcribe, HTML motion graphics
HeyGen examples Avatar + TTS pipeline
OpenRouterProvider Chat completions only

5. Catalog sync sources (for issue #08)

Source Endpoint / method
OpenRouter GET https://openrouter.ai/api/v1/videos/models
Replicate Per-model OpenAPI schema + pricing on model page
Picker manual rows Alibaba direct $/s, Vidu off-peak, workflow $0

6. Summary metrics

Metric Value
LiteLLM-native families 5
Picker routes (approx.) 30+
Supported end-to-end today 4 cloud families + 0 workflow tools packaged
Planned provider adapters OpenRouter, Replicate, fal, Vidu, DashScope
Desktop video API in open engine None (issue #09)
主要言語
Python
スター
9.3k
フォーク
1.5k
平均マージ
1日 5時間
マージ済み PR(30日)
520

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

MervinPraison/PraisonAI のほかの issue

MervinPraison/PraisonAI の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。