[epic][video] Cost Picker parity — multi-provider video generation

Aperta
#5,171 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
18/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Attiva
Stack tecnologico
python

Direzione di ricerca

This is an epic with no single change to make. Start by reading src/praisonai-agents/praisonaiagents/agent/video_agent.py and src/praisonai/praisonai/capabilities/videos.py to see how VideoAgent.generate() currently routes through LiteLLM. Done means one child deliverable (such as the provider registry, #02) is agreed with maintainers and merged with mocked HTTP tests; the epic itself is not a starter task.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

documentation javascript origin/outsider performance security

Executive summary

The Video Model Cost Picker is a decision support tool for music-video and short-clip production: it compares 30+ routes spanning zero-cost ffmpeg/ComfyUI workflows, Replicate predictions, fal.ai queues, OpenRouter video jobs, direct Vidu/Alibaba APIs, and first-party OpenAI/Google/Runway APIs.

PraisonAI Python VideoAgent currently generates video only through LiteLLM, which implements five upstream families: OpenAI Sora, Azure Sora, Gemini Veo 3.1, Vertex Veo 3.1, RunwayML. Every other picker row is unsupported in code. OpenRouter integration today is chat-only (praisonai.llm.gateways.OpenRouterProvider). Replicate is LLM examples only. Desktop reportedly exposes a single Replicate video model while the open praisonai-desktop engine has no video HTTP API.

This epic delivers provider registry + four cloud adapters + workflow tools + pricing catalog + desktop UI so agents, CLI, MCP, and desktop can execute the same routes users already priced in the picker.


Baseline architecture (as shipped)

┌─────────────┐  ┌──────────────┐  ┌─────────────────┐
│ YAML/Agent  │  │ CLI videos   │  │ MCP videos.gen  │
└──────┬──────┘  └──────┬───────┘  └────────┬────────┘
       │                │                    │
       └────────────────┼────────────────────┘
                        v
              VideoAgent.generate()
                        v
              litellm.videos.video_generation
                        v
     openai | azure | gemini | vertex_ai | runwayml

Primary source files:

Component Path
VideoAgent src/praisonai-agents/praisonaiagents/agent/video_agent.py
VideoConfig same file — seconds, size, poll_interval, max_wait_time, timeout
Capabilities src/praisonai/praisonai/capabilities/videos.py
CLI src/praisonai-code/praisonai_code/cli/features/capabilities.py → handle_videos
MCP src/praisonai-mcp/praisonai_mcp/mcp_server/adapters/extended_capabilities.py
Unit tests src/praisonai-agents/tests/test_video_agent.py

LiteLLM-supported llm= values (documented):

  • openai/sora-2, openai/sora-2-pro
  • azure/sora-2, azure/sora-2-pro
  • gemini/veo-3.1-generate-preview, -fast-, -lite-
  • vertex_ai/veo-3.1-generate-001, -fast-, -lite-
  • runwayml/gen4_turbo

Target architecture

VideoAgent / video_generate / CLI / MCP / Desktop engine
                        │
                        v
           VideoProviderRegistry.resolve(model)
                        │
     ┌──────────────────┼──────────────────┬─────────────┐
     v                  v                  v             v
LiteLLMAdapter   OpenRouterProvider  ReplicateProvider  FalProvider
     │                  │                  │             │
     v                  v                  v             v
  Sora/Veo/Runway   OR /videos API    Predictions API   fal queue
                        │
                   ViduProvider + DashScopeWanProvider
                        │
              Workflow tools (ffmpeg / ComfyUI) — not providers

Companion matrix: paste SUPPORT-MATRIX.md into wiki for route-by-route status.


Child issues and technical scope

# Issue Technical deliverable
02 Provider registry VideoProvider protocol, namespaced job ids, unify 5 entry points
03 OpenRouter video REST client: POST/GET /videos, download content; header parity with chat
04 Replicate video Predictions lifecycle, per-model input profiles, MiniMax preset
05 fal video Queue API, LTX 6s min, Omni two-frame
06 Vidu + DashScope Direct APIs, off-peak queue, KF2V 5s fixed
07 ffmpeg / ComfyUI Tools: ambient-loop, composite-freeze, LTX guide/API
08 Cost catalog JSON schema, sync script, estimate_job, CLI compare
09 Desktop /video/* engine routes + UI picker
10 Docs Matrix, examples, CI guard, deprecation table

Phased rollout

Phase 1 — Foundation (blocks everything)
  • #02 registry + LiteLLM adapter extraction
  • #10 matrix skeleton (all rows planned)
  • #08 catalog static JSON (picker snapshot)
Phase 2 — Cloud channels (parallel)
  • #03 OpenRouter (largest SKU coverage)
  • #04 Replicate (desktop + cheap routes)
  • #05 fal
  • #06 direct vendors
Phase 3 — Product
  • #07 workflows
  • #09 desktop
  • #08 estimator UI wiring

Epic acceptance criteria

  • Coverage: ≥3 picker “cheap paid” routes + 3 workflow/free routes supported with mocked e2e tests
  • Regression: Sora/Veo/Runway examples unchanged behavior
  • Matrix: 100% picker routes documented with status
  • Credentials: .env.example complete; onboarding mentions video keys
  • Desktop: Multi-model picker when keys configured
  • Estimator: Default 8 clips × 5s scenario reproduces picker job costs within 1% on fixture catalog

Risks

Risk Mitigation
LiteLLM adds OpenRouter/Replicate video Wrap in adapter; don’t fork vendor HTTP in two places
Replicate schema drift Version-pinned profiles + catalog version bump
Long Vidu off-peak jobs Async job docs; don’t use 600s default wait
TS/Rust parity expectations #10 clarifies Python-only generation scope

Out of scope

  • Full Google Flow product clone
  • HeyGen avatar pipeline (existing examples)
  • TypeScript VideoAgent generation unless follow-up epic

References


Appendix: Stakeholders and success metrics

  • Agent developers can run picker routes via VideoAgent without custom HTTP code.
  • Desktop users select models from catalog, not a single hardcoded Replicate slug.
  • Finance/creative leads can estimate batch cost before generation (#08).
  • Support team has single matrix (#10) for 'is route X supported?'

Appendix: Weekly milestone sketch

  1. Week 1–2: Registry (#02) + matrix skeleton (#10) + catalog JSON (#08).
  2. Week 3–4: OpenRouter (#03) + Replicate phase-1 (#04).
  3. Week 5: fal (#05) + DashScope/Vidu spike (#06).
  4. Week 6: ffmpeg workflows (#07) + desktop engine API (#09).
  5. Week 7: Hardening, docs, CI guards, beta desktop UI.

Appendix: Observability (epic-wide)

  • Emit structured log fields: provider, model, job_id, duration_ms, status.
  • Optional OpenTelemetry span: video.generate → video.poll → video.download.
  • Counter metrics: praisonai_video_jobs_total{provider,status}.
  • Histogram: praisonai_video_job_duration_seconds{provider}.

Appendix: Security (epic-wide)

  • Never log raw API keys, Authorization headers, or full prompts at INFO level.
  • Redact secrets in desktop engine logs (see engine/test_settings_redaction.py).
  • Validate user-supplied URLs in Replicate/fal download steps to prevent SSRF (allowlist replicate.delivery, fal.media domains).
  • Cap upload size for input_reference images (e.g. 20 MB) before provider upload.
  • Use HTTPS only for all provider base URLs; reject http:// overrides unless explicit dev flag.

Appendix: Glossary

  • Route — Named row in cost picker (e.g. p-video draft).
  • Channel — Billing path: OpenRouter, Replicate, fal, direct, LiteLLM-native.
  • Provider adapter — Python class implementing VideoProvider protocol.
  • Workflow — ffmpeg/ComfyUI tool; not a cloud VideoProvider.
  • Catalog — JSON list of routes, pricing, capabilities (#08).

Appendix: Epic-level definition of done checklist

  1. All child issues closed or explicitly deferred with matrix status not_planned.
  2. CI green on mocked e2e for 3 cloud + 2 workflow routes.
  3. No regression in tests/test_video_agent.py for LiteLLM mocks.
  4. Published docs linked from README.
  5. Desktop beta can save MP4 from ≥3 providers when keys configured.







Cross-issue dependency matrix

From To Contract
#02 #03 OpenRouterVideoProvider registered
#02 #04 ReplicateVideoProvider registered
#08 #03 Catalog sync from GET /videos/models
#09 #02 Engine calls VideoAgent only
#10 all Matrix status updated on merge

Quality gates (CI)

  • pytest src/praisonai-agents/tests/video/
  • pytest src/praisonai-desktop/engine/test_video_routes.py
  • Docs link checker for docs/video/SUPPORT_MATRIX.md
  • Optional live smoke job gated by PRAISONAI_LIVE_VIDEO=1

Release criteria

  • All provider adapters merged with mocked HTTP tests green.
  • SUPPORT_MATRIX.md lists every picker route with status.
  • video_routes_catalog.json validates against JSON Schema in CI.
  • Desktop engine exposes /video/generate behind existing auth gate.
  • Examples exist for OpenRouter, Replicate, and fal minimum routes.
  • Workflow tools ambient-loop and composite-freeze callable as @tool.
  • No secrets in logs per engine redaction tests.
  • CLI videos estimate subcommand returns JSON for 8×5 default scenario.
  • MCP videos.generate accepts openrouter/ and replicate/ model prefixes.
  • Documentation clarifies TS VideoAgent is analysis-only.
  • Deprecation table published for Veo 3.0 and runway-gen3 MCP default.
  • CHANGELOG entry summarizes breaking registry job id format.
  • Performance: poll loop respects VideoConfig.max_wait_time.
  • Security: SSRF allowlist enforced on provider download URLs.
  • User-facing INSTALL documents required env vars per provider.

Picker route coverage targets

Tier Routes Target count
Tier 0 workflow ambient-loop, composite, LTX doc 3
Tier 1 cheap cloud p-video draft, Wan 2.2 i2v, LTX fal 3
Tier 2 OpenRouter Kling 3, Hailuo, Seedance 3
Tier 3 premium Veo OR resold, Sora, Runway 3

Epic communication checklist

  1. Post matrix link in #video channel when #10 skeleton merges.
  2. Weekly demo: one new picker route end-to-end per sprint.
  3. Track deferred routes with not_planned + reason in matrix.
  4. Align desktop beta timeline with #09 engine API freeze.
  5. Coordinate breaking job id change with downstream recipe examples.

Additional verification notes

  • Validate epic acceptance against SUPPORT_MATRIX row count equals picker route count.
  • Confirm OpenRouter catalog sync job runs without OPENROUTER_API_KEY in CI (uses fixture).
  • Confirm Replicate profiles pin version hashes in catalog JSON.
  • Verify desktop engine never stores MP4 in logs-only temp without user consent path.
  • Run regression: YAML workflow VideoAgent smoke with mocked LiteLLM adapter.
  • Document Flow vs Veo API billing difference in BILLING_CAVEATS.md (#10).
  • Ensure MCP tool schema documents seconds as string matching VideoConfig.
  • Ensure CLI videos command accepts namespaced job id on status subcommand (future).
  • Review child issue order in INDEX before filing on GitHub.
  • Confirm LiteLLM adapter remains default when model has no recognized prefix.
  • Add epic label and project board column for video parity tracking.
  • Verify examples/python/video/README lists minimum env vars per provider.
  • Check TS/Rust VideoAgent docs explicitly state analysis-only scope.
  • Run support matrix diff in PR template for any new picker route.
  • Schedule security review for SSRF allowlist before desktop beta.

Reference artifact: Video Model Cost Picker

Use this document as the epic body or project wiki appendix.


1. Runtime stack (Python generation)

1.1 VideoAgent call chain
Step Implementation
Construct VideoAgent(llm=..., video=VideoConfig(seconds="8", size="1280x720", poll_interval=10, max_wait_time=600, timeout=600))
Generate generate(prompt, seconds=, size=, input_reference=, **kwargs)
Poll wait_for_completion(video_id) — loops status() until completed or failed
Download content(video_id) → bytes; or download() helper if present on agent

Source: src/praisonai-agents/praisonaiagents/agent/video_agent.py

1.2 LiteLLM functions invoked
Method LiteLLM callable
Sync generate litellm.videos.video_generation
Async generate litellm.videos.avideo_generation
Status video_status / avideo_status
Bytes video_content / avideo_content
List video_list / avideo_list
Remix video_remix / avideo_remix
1.3 LiteLLM video provider implementations (upstream)
Prefix Backend module pattern
openai/ litellm/llms/openai/videos/transformation.py
azure/ litellm/llms/azure/videos/transformation.py
gemini/ litellm/llms/gemini/videos/transformation.py
vertex_ai/ litellm/llms/vertex_ai/videos/transformation.py
runwayml/ litellm/llms/runwayml/videos/transformation.py

No upstream LiteLLM video handler for: replicate/, openrouter/, fal/, vidu/, dashscope/.

1.4 Parallel entry points (must stay consistent after registry #02)
API Module Default model
SDK VideoAgent openai/sora-2
Capability praisonai.capabilities.video_generate runway/gen3a_turbo
CLI praisonai videos sora
MCP praisonai.videos.generate runway-gen3

2. Supported model strings today (LiteLLM path)

llm= Env vars Picker analog
openai/sora-2 OPENAI_API_KEY sora_2
openai/sora-2-pro OPENAI_API_KEY sora_2_pro
azure/sora-2 Azure OpenAI vars Azure Sora
gemini/veo-3.1-generate-preview GOOGLE_API_KEY / GEMINI_API_KEY Veo 3.1 full (partial channel)
gemini/veo-3.1-fast-generate-preview same Veo 3.1 Fast
gemini/veo-3.1-lite-generate-preview same Veo 3.1 Lite
vertex_ai/veo-3.1-generate-001 GCP credentials Veo 3.1 Vertex silent
vertex_ai/veo-3.1-fast-generate-001 GCP Veo 3.1 Fast
vertex_ai/veo-3.1-lite-generate-001 GCP Veo 3.1 Lite
runwayml/gen4_turbo Runway API secret Runway Gen-4 (not Gen-4.5 Replicate listing)

Veo kwargs (LiteLLM passthrough): last_frame, reference_images for 3.1 interpolation.


3. Cost picker routes — full status table

Status codes: S Supported · P Partial (family only) · M Missing · W Workflow (offline tool)

Route Channel in picker Status PraisonAI target (llm= / tool)
ambient-loop ffmpeg W tool:video_ambient_loop (#07)
Local composite ffmpeg + mask M→W tool:video_composite_freeze_background (#07)
LTX Cinemagraph LoRA ComfyUI W docs + optional ComfyUI API (#07)
MiniMax H3 Max free fal M fal/minimax-h3-max (#05)
LTX-2.3 Fast fal M fal/ltx-2.3-fast (#05)
p-video (draft) Replicate M replicate/... + draft:true (#04)
p-video Replicate M replicate/... (#04)
Wan 2.2 i2v Fast Replicate M flat $0.05/run (#04)
Seedance 1.5 Pro Replicate / OR M generate_audio:false (#04/#03)
Luma Ray 3.2 Replicate M loop:true, not with end_image (#04)
Vidu Q3 Turbo Vidu M vidu/q3-turbo (#06)
Wan 2.6 Flash Alibaba M dashscope/wan2.6-flash (#06)
Kling 2.1 Replicate M end_image + mode=pro (#04)
Wan 1.3B inpaint Replicate M mask schema (#04)
Veo 3.1 Lite Vertex / OR P / M S: gemini/vertex lite; M: OpenRouter resell (#03)
Wan 2.2 KF2V Flash Alibaba M 5s fixed (#06)
MiniMax Hailuo 3 Max OpenRouter M openrouter/... (#03)
Wan VACE 14B inpaint Replicate M src_mask (#04)
Kling 3.0 Std / Pro OpenRouter M (#03)
Wan 2.7 OR / Alibaba M (#03/#06)
Runway Gen-4.5 Runway / Replicate P / M P: runwayml/gen4_turbo; M: Replicate SKU (#04)
Veo 3.1 Fast Vertex / OR P / M (#03 + gemini fast)
MiniMax Hailuo 3 OpenRouter M (#03)
Wan VACE 14B Replicate M (#04)
Gemini Omni Flash 1.1 Gemini + fal M (#05 + future Gemini API)
Wan 3.0 OpenRouter M keyframe warning (#03)
Veo 3.1 full Vertex / OR P / M gemini/vertex standard (#03)
FLUX 3 Video OpenRouter M (#03)
Seedance 2.0 OpenRouter M token billing (#03)
MiniMax video-01 Replicate M desktop preset (#04/#09)

4. Non-generation video code (do not confuse with matrix)

Package Behavior
praisonai-ts VideoAgent Multimodal analysis; default llm gpt-4o
praisonai-rust VideoAgent Stub generate()
praisonai_tools.video ffmpeg edit, transcribe, HTML motion graphics
HeyGen examples Avatar + TTS pipeline
OpenRouterProvider Chat completions only

5. Catalog sync sources (for issue #08)

Source Endpoint / method
OpenRouter GET https://openrouter.ai/api/v1/videos/models
Replicate Per-model OpenAPI schema + pricing on model page
Picker manual rows Alibaba direct $/s, Vidu off-peak, workflow $0

6. Summary metrics

Metric Value
LiteLLM-native families 5
Picker routes (approx.) 30+
Supported end-to-end today 4 cloud families + 0 workflow tools packaged
Planned provider adapters OpenRouter, Replicate, fal, Vidu, DashScope
Desktop video API in open engine None (issue #09)
Lingua principale
Python
Stelle
9.3k
Fork
1.5k
Merge medio
1g 5h
PR unite (30g)
520

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di MervinPraison/PraisonAI

Tutte le issue di MervinPraison/PraisonAI

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.