[video] Zero-cost ffmpeg and ComfyUI cinemagraph workflows
Los mantenedores suelen responder en 1 día
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 22/100
Línea de trabajo
Start with the existing ffmpeg helpers in src/praisonai-browser/praisonai_browser/video.py (create_video_ffmpeg, FFmpegVideoEncoder) and the render pipeline in praisonai_tools/video/render.py. Pick one workflow, such as video_ambient_loop, and build it as a single @tool function, checking the output loop point with ffprobe. Done means a tiny 64×64 fixture plate produces a seamless MP4 in a test marked requires_ffmpeg. The composite and ComfyUI workflows depend on separate design decisions, so they should be split into their own issues.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Epic: #5171
Problem statement
The Video Model Cost Picker cheapest routes are $0.00 job cost and do not call cloud video APIs:
| Route | Mechanism | Picker constraints |
|---|---|---|
| ambient-loop | ffmpeg displacement field on plate | Loops natively; good for water/haze/cloth; bad for directional leaf rustle |
| Local composite | Cloud gen (any model) + segmentation mask + ffmpeg composite | Freezes background/faces when T2V re-poses subjects |
| LTX Cinemagraph LoRA | ComfyUI + LoRA on Apple Silicon | One element moves; ~3 min/clip @ 512×704 M1 Max; MASK INPUT |
PraisonAI already uses ffmpeg in:
src/praisonai-browser/praisonai_browser/video.py—create_video_ffmpeg,FFmpegVideoEncoderpraisonai_tools/video/render.py— edit pipeline rendering
No unified agent-invokable tools implement picker route names.
Architecture: workflows vs providers
These are not VideoProvider registry entries (#02). Implement as:
praisonaiagents/tools/video_workflows.py # @tool functions
# OR
praisonai_tools/video/workflows/
ambient_loop.py
composite_freeze.py
comfyui_ltx.py
YAML agents call tools by name; cost catalog (#08) marks provider: "workflow", job_cost_usd: 0 for ffmpeg-only steps.
Workflow 1: ambient-loop
Input
plate_path: PNG/JPG loop-friendly texture (water, haze, cloth)duration_seconds: output length (picker default 5s per clip)fps: default 24 or 30- Optional
displacement_strength,period_secondstuning
Processing
- Build ffmpeg
filter_complexwith periodic displacement (implementation from picker methodology — sine/cosine displacement maps). - Output seamless loop MP4 (verify loop point with
ffprobein test).
Tool signature (proposed)
@tool
def video_ambient_loop(
plate_path: str,
output_path: str,
duration_seconds: float = 5.0,
fps: int = 24,
) -> str:
"""Create seamless ambient loop video from still plate via ffmpeg displacement."""
Failure modes
- ffmpeg missing on PATH → clear install message (link ffmpeg.org)
- Non-loop-friendly content → document picker limitation in docstring
Workflow 2: composite-freeze (Local composite)
Input
generated_video_path: MP4 from anyVideoAgentproviderreference_still_path: optional; default first frame of videomask: optional user mask; else auto segment with MediaPipe Selfie Segmentation or person mask model
Processing pipeline
1. Extract reference still (frame 0)
2. Compute mask per frame OR static mask from reference
3. ffmpeg: for each frame, composite still outside mask over generated frame
4. Encode output MP4 (H.264)
Tool signature
@tool
def video_composite_freeze_background(
generated_video_path: str,
output_path: str,
mask_expand_px: int = 4,
use_mediapipe: bool = True,
) -> str:
Dependencies
ffmpegrequiredmediapipeoptional extra:pip install praisonaiagents[video-composite]or document manual install
Picker alignment
Addresses “THE FIX” in artifact: models re-pose people; freezing background via mask is $0 incremental cost in picker table.
Workflow 3: LTX Cinemagraph LoRA (ComfyUI)
Scope
Phase A (required): Documentation + workflow JSON export path:
- Hardware: Apple M-series recommended (picker)
- Resolution: 512×704
- LoRA: LTX Cinemagraph (distilled GGUF Q4 referenced in picker)
- Expected runtime ~3 min/clip on M1 Max
Phase B (optional): ComfyUI API bridge
@tool
def video_comfyui_ltx_cinemagraph(
input_image_path: str,
output_path: str,
comfyui_base_url: str = "http://127.0.0.1:8188",
workflow_json_path: str = "...",
) -> str:
Submit prompt via ComfyUI /prompt API; poll /history/{prompt_id}; download output video node.
Do not ship ComfyUI or LoRA weights in PraisonAI wheel.
Agent orchestration pattern
Example team YAML logic (documentation):
- If scene tags ∈
{water, haze, cloth}→ callvideo_ambient_loop - Else
VideoAgent.generatewith cheapest supported cloud route from catalog (#08) - If
evaluate_motion_on_facesdetects drift →video_composite_freeze_background - If user needs single-element loop → ComfyUI doc path
Cost estimator (#08)
Catalog entries:
"id": "ambient-loop",
"pricing": { "mode": "free" },
"flags": ["FREE", "LOOPS NATIVELY"],
"supported": true,
"implementation": "tool:video_ambient_loop"
Composite workflow: cloud_cost + 0 for ffmpeg step.
Testing
| Test | Marker |
|---|---|
| ambient-loop 1s output | @pytest.mark.requires_ffmpeg |
| composite with synthetic 10-frame MP4 + fixed mask | skip mediapipe in CI |
| ComfyUI bridge | @pytest.mark.integration mock HTTP |
Use tiny fixtures in tests/fixtures/video/ (64×64 plate PNG).
Acceptance criteria
-
video_ambient_looptool + documented CLI equivalent -
video_composite_freeze_backgroundacceptsVideoAgentoutput path - ComfyUI guide in
docs/video/LTX_CINEMAGRAPH.mdwith graph export instructions - Matrix: three picker rows Supported (workflow)
- Agent example in
examples/python/video/workflows/
Dependencies
Cloud generation optional for composite step (#02–#06)
Related
Epic #01, #08, #10
References
praisonai_tools/video/— probe, render, motion_graphicspraisonai_browser/video.py— ffmpeg helpers
Appendix: ffmpeg filter notes
- ambient-loop: periodic displacement; verify loop point with ffprobe
duration. - composite: blend masked regions using
alphamerge+overlaychain. - Use yuv420p pixel format for compatibility with players.
Appendix: Tool test matrix
| Test | Marker |
|---|---|
| test_ambient_1 | requires_ffmpeg |
| test_ambient_2 | requires_ffmpeg |
| test_ambient_3 | requires_ffmpeg |
| test_ambient_4 | requires_ffmpeg |
| test_ambient_5 | requires_ffmpeg |
| test_ambient_6 | requires_ffmpeg |
| test_ambient_7 | requires_ffmpeg |
| test_ambient_8 | requires_ffmpeg |
| test_ambient_9 | requires_ffmpeg |
| test_ambient_10 | requires_ffmpeg |
| test_ambient_11 | requires_ffmpeg |
| test_ambient_12 | requires_ffmpeg |
| test_ambient_13 | requires_ffmpeg |
| test_ambient_14 | requires_ffmpeg |
| test_ambient_15 | requires_ffmpeg |
| test_ambient_16 | requires_ffmpeg |
| test_ambient_17 | requires_ffmpeg |
| test_ambient_18 | requires_ffmpeg |
ffmpeg CLI examples
ffmpeg -version # prerequisite
# ambient-loop: see docs/video/workflows/ambient-loop.md
# composite-freeze: see docs/video/workflows/composite-freeze.md
Workflow integration tests (enumeration)
| Test id | Scenario | Fixture | Expected |
|---|---|---|---|
test_workflow_01 |
Scenario 1 | mock HTTP | pass |
test_workflow_02 |
Scenario 2 | mock HTTP | pass |
test_workflow_03 |
Scenario 3 | mock HTTP | pass |
test_workflow_04 |
Scenario 4 | mock HTTP | pass |
test_workflow_05 |
Scenario 5 | mock HTTP | pass |
test_workflow_06 |
Scenario 6 | mock HTTP | pass |
test_workflow_07 |
Scenario 7 | mock HTTP | pass |
test_workflow_08 |
Scenario 8 | mock HTTP | pass |
test_workflow_09 |
Scenario 9 | mock HTTP | pass |
test_workflow_10 |
Scenario 10 | mock HTTP | pass |
test_workflow_11 |
Scenario 11 | mock HTTP | pass |
test_workflow_12 |
Scenario 12 | mock HTTP | pass |
test_workflow_13 |
Scenario 13 | mock HTTP | pass |
test_workflow_14 |
Scenario 14 | mock HTTP | pass |
test_workflow_15 |
Scenario 15 | mock HTTP | pass |
test_workflow_16 |
Scenario 16 | mock HTTP | pass |
test_workflow_17 |
Scenario 17 | mock HTTP | pass |
test_workflow_18 |
Scenario 18 | mock HTTP | pass |
test_workflow_19 |
Scenario 19 | mock HTTP | pass |
test_workflow_20 |
Scenario 20 | mock HTTP | pass |
test_workflow_21 |
Scenario 21 | mock HTTP | pass |
test_workflow_22 |
Scenario 22 | mock HTTP | pass |
test_workflow_23 |
Scenario 23 | mock HTTP | pass |
test_workflow_24 |
Scenario 24 | mock HTTP | pass |
test_workflow_25 |
Scenario 25 | mock HTTP | pass |
test_workflow_26 |
Scenario 26 | mock HTTP | pass |
test_workflow_27 |
Scenario 27 | mock HTTP | pass |
test_workflow_28 |
Scenario 28 | mock HTTP | pass |
Agent tool contracts
| Tool | Input | Output |
|---|---|---|
| video_ambient_loop | plate_path, output_path | MP4 path |
| video_composite_freeze_background | generated_mp4, output_path | MP4 path |
| video_comfyui_ltx_cinemagraph | image, comfy URL | MP4 path or doc link |
- Lenguaje dominante
- Python
- Estrellas
- 9.3k
- Forks
- 1.5k
- Merge medio
- 1 d 5 h
- PR fusionados (30 d)
- 520
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de MervinPraison/PraisonAI
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
MervinPraison/PraisonAI#6710 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
claude
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
MervinPraison/PraisonAI#6707 ·
Los mantenedores suelen responder en 1 día
-
bug claude
Dificultad 1/5 Menos de una hora Aptitud para principiantes 90/100
MervinPraison/PraisonAI#6671 ·
Los mantenedores suelen responder en 1 día
-
Compaction budget regression fixture remains below budget with accurate tokenizationQuizá libre de nuevo Un pull request para esta issue se cerró sin fusionarse. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
MervinPraison/PraisonAI#5418 · 3 comentarios ·
Los mantenedores suelen responder en 1 día
-
bug claude documentation
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
MervinPraison/PraisonAI#5341 · 3 comentarios ·
Los mantenedores suelen responder en 1 día
Todos los issues de MervinPraison/PraisonAI
Issues similares
-
cognition
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
Los mantenedores suelen responder en 1 día
-
[Suggestion]: Geonode MCP ServerAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 78/100
collabnix/awesome-mcp-lists#433 ·
Los mantenedores suelen responder en 4 días
-
CrossEntropyFilter raises on zero probabilities and single-class resultsPosiblemente ocupada @NopeCharon la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
datajoint/datajoint-python#1703 ·
Los mantenedores suelen responder en 6 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Archmonger/ServeStatic#129 ·