[video] Zero-cost ffmpeg and ComfyUI cinemagraph workflows

Abierto
#5,177 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
22/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
ai, tooling

Línea de trabajo

Start with the existing ffmpeg helpers in src/praisonai-browser/praisonai_browser/video.py (create_video_ffmpeg, FFmpegVideoEncoder) and the render pipeline in praisonai_tools/video/render.py. Pick one workflow, such as video_ambient_loop, and build it as a single @tool function, checking the output loop point with ffprobe. Done means a tiny 64×64 fixture plate produces a seamless MP4 in a test marked requires_ffmpeg. The composite and ComfyUI workflows depend on separate design decisions, so they should be split into their own issues.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

documentation origin/outsider

Epic: #5171


Problem statement

The Video Model Cost Picker cheapest routes are $0.00 job cost and do not call cloud video APIs:

Route Mechanism Picker constraints
ambient-loop ffmpeg displacement field on plate Loops natively; good for water/haze/cloth; bad for directional leaf rustle
Local composite Cloud gen (any model) + segmentation mask + ffmpeg composite Freezes background/faces when T2V re-poses subjects
LTX Cinemagraph LoRA ComfyUI + LoRA on Apple Silicon One element moves; ~3 min/clip @ 512×704 M1 Max; MASK INPUT

PraisonAI already uses ffmpeg in:

  • src/praisonai-browser/praisonai_browser/video.py — create_video_ffmpeg, FFmpegVideoEncoder
  • praisonai_tools/video/render.py — edit pipeline rendering

No unified agent-invokable tools implement picker route names.


Architecture: workflows vs providers

These are not VideoProvider registry entries (#02). Implement as:

praisonaiagents/tools/video_workflows.py   # @tool functions
# OR
praisonai_tools/video/workflows/
  ambient_loop.py
  composite_freeze.py
  comfyui_ltx.py

YAML agents call tools by name; cost catalog (#08) marks provider: "workflow", job_cost_usd: 0 for ffmpeg-only steps.


Workflow 1: ambient-loop

Input
  • plate_path: PNG/JPG loop-friendly texture (water, haze, cloth)
  • duration_seconds: output length (picker default 5s per clip)
  • fps: default 24 or 30
  • Optional displacement_strength, period_seconds tuning
Processing
  1. Build ffmpeg filter_complex with periodic displacement (implementation from picker methodology — sine/cosine displacement maps).
  2. Output seamless loop MP4 (verify loop point with ffprobe in test).
Tool signature (proposed)
@tool
def video_ambient_loop(
    plate_path: str,
    output_path: str,
    duration_seconds: float = 5.0,
    fps: int = 24,
) -> str:
    """Create seamless ambient loop video from still plate via ffmpeg displacement."""
Failure modes
  • ffmpeg missing on PATH → clear install message (link ffmpeg.org)
  • Non-loop-friendly content → document picker limitation in docstring

Workflow 2: composite-freeze (Local composite)

Input
  • generated_video_path: MP4 from any VideoAgent provider
  • reference_still_path: optional; default first frame of video
  • mask: optional user mask; else auto segment with MediaPipe Selfie Segmentation or person mask model
Processing pipeline
1. Extract reference still (frame 0)
2. Compute mask per frame OR static mask from reference
3. ffmpeg: for each frame, composite still outside mask over generated frame
4. Encode output MP4 (H.264)
Tool signature
@tool
def video_composite_freeze_background(
    generated_video_path: str,
    output_path: str,
    mask_expand_px: int = 4,
    use_mediapipe: bool = True,
) -> str:
Dependencies
  • ffmpeg required
  • mediapipe optional extra: pip install praisonaiagents[video-composite] or document manual install
Picker alignment

Addresses “THE FIX” in artifact: models re-pose people; freezing background via mask is $0 incremental cost in picker table.


Workflow 3: LTX Cinemagraph LoRA (ComfyUI)

Scope

Phase A (required): Documentation + workflow JSON export path:

  • Hardware: Apple M-series recommended (picker)
  • Resolution: 512×704
  • LoRA: LTX Cinemagraph (distilled GGUF Q4 referenced in picker)
  • Expected runtime ~3 min/clip on M1 Max

Phase B (optional): ComfyUI API bridge

@tool
def video_comfyui_ltx_cinemagraph(
    input_image_path: str,
    output_path: str,
    comfyui_base_url: str = "http://127.0.0.1:8188",
    workflow_json_path: str = "...",
) -> str:

Submit prompt via ComfyUI /prompt API; poll /history/{prompt_id}; download output video node.

Do not ship ComfyUI or LoRA weights in PraisonAI wheel.


Agent orchestration pattern

Example team YAML logic (documentation):

  1. If scene tags ∈ {water, haze, cloth} → call video_ambient_loop
  2. Else VideoAgent.generate with cheapest supported cloud route from catalog (#08)
  3. If evaluate_motion_on_faces detects drift → video_composite_freeze_background
  4. If user needs single-element loop → ComfyUI doc path

Cost estimator (#08)

Catalog entries:

  "id": "ambient-loop",
  "pricing": { "mode": "free" },
  "flags": ["FREE", "LOOPS NATIVELY"],
  "supported": true,
  "implementation": "tool:video_ambient_loop"

Composite workflow: cloud_cost + 0 for ffmpeg step.


Testing

Test Marker
ambient-loop 1s output @pytest.mark.requires_ffmpeg
composite with synthetic 10-frame MP4 + fixed mask skip mediapipe in CI
ComfyUI bridge @pytest.mark.integration mock HTTP

Use tiny fixtures in tests/fixtures/video/ (64×64 plate PNG).


Acceptance criteria

  • video_ambient_loop tool + documented CLI equivalent
  • video_composite_freeze_background accepts VideoAgent output path
  • ComfyUI guide in docs/video/LTX_CINEMAGRAPH.md with graph export instructions
  • Matrix: three picker rows Supported (workflow)
  • Agent example in examples/python/video/workflows/

Dependencies

Cloud generation optional for composite step (#02–#06)

Related

Epic #01, #08, #10

References

  • praisonai_tools/video/ — probe, render, motion_graphics
  • praisonai_browser/video.py — ffmpeg helpers

Appendix: ffmpeg filter notes

  • ambient-loop: periodic displacement; verify loop point with ffprobe duration.
  • composite: blend masked regions using alphamerge + overlay chain.
  • Use yuv420p pixel format for compatibility with players.

Appendix: Tool test matrix

Test Marker
test_ambient_1 requires_ffmpeg
test_ambient_2 requires_ffmpeg
test_ambient_3 requires_ffmpeg
test_ambient_4 requires_ffmpeg
test_ambient_5 requires_ffmpeg
test_ambient_6 requires_ffmpeg
test_ambient_7 requires_ffmpeg
test_ambient_8 requires_ffmpeg
test_ambient_9 requires_ffmpeg
test_ambient_10 requires_ffmpeg
test_ambient_11 requires_ffmpeg
test_ambient_12 requires_ffmpeg
test_ambient_13 requires_ffmpeg
test_ambient_14 requires_ffmpeg
test_ambient_15 requires_ffmpeg
test_ambient_16 requires_ffmpeg
test_ambient_17 requires_ffmpeg
test_ambient_18 requires_ffmpeg







ffmpeg CLI examples

ffmpeg -version  # prerequisite
# ambient-loop: see docs/video/workflows/ambient-loop.md
# composite-freeze: see docs/video/workflows/composite-freeze.md

Workflow integration tests (enumeration)

Test id Scenario Fixture Expected
test_workflow_01 Scenario 1 mock HTTP pass
test_workflow_02 Scenario 2 mock HTTP pass
test_workflow_03 Scenario 3 mock HTTP pass
test_workflow_04 Scenario 4 mock HTTP pass
test_workflow_05 Scenario 5 mock HTTP pass
test_workflow_06 Scenario 6 mock HTTP pass
test_workflow_07 Scenario 7 mock HTTP pass
test_workflow_08 Scenario 8 mock HTTP pass
test_workflow_09 Scenario 9 mock HTTP pass
test_workflow_10 Scenario 10 mock HTTP pass
test_workflow_11 Scenario 11 mock HTTP pass
test_workflow_12 Scenario 12 mock HTTP pass
test_workflow_13 Scenario 13 mock HTTP pass
test_workflow_14 Scenario 14 mock HTTP pass
test_workflow_15 Scenario 15 mock HTTP pass
test_workflow_16 Scenario 16 mock HTTP pass
test_workflow_17 Scenario 17 mock HTTP pass
test_workflow_18 Scenario 18 mock HTTP pass
test_workflow_19 Scenario 19 mock HTTP pass
test_workflow_20 Scenario 20 mock HTTP pass
test_workflow_21 Scenario 21 mock HTTP pass
test_workflow_22 Scenario 22 mock HTTP pass
test_workflow_23 Scenario 23 mock HTTP pass
test_workflow_24 Scenario 24 mock HTTP pass
test_workflow_25 Scenario 25 mock HTTP pass
test_workflow_26 Scenario 26 mock HTTP pass
test_workflow_27 Scenario 27 mock HTTP pass
test_workflow_28 Scenario 28 mock HTTP pass

Agent tool contracts

Tool Input Output
video_ambient_loop plate_path, output_path MP4 path
video_composite_freeze_background generated_mp4, output_path MP4 path
video_comfyui_ltx_cinemagraph image, comfy URL MP4 path or doc link
Lenguaje dominante
Python
Estrellas
9.3k
Forks
1.5k
Merge medio
1 d 5 h
PR fusionados (30 d)
520

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de MervinPraison/PraisonAI

Todos los issues de MervinPraison/PraisonAI

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.