[video] Zero-cost ffmpeg and ComfyUI cinemagraph workflows

Aperta
#5,177 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
22/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, tooling

Direzione di ricerca

Start with the existing ffmpeg helpers in src/praisonai-browser/praisonai_browser/video.py (create_video_ffmpeg, FFmpegVideoEncoder) and the render pipeline in praisonai_tools/video/render.py. Pick one workflow, such as video_ambient_loop, and build it as a single @tool function, checking the output loop point with ffprobe. Done means a tiny 64×64 fixture plate produces a seamless MP4 in a test marked requires_ffmpeg. The composite and ComfyUI workflows depend on separate design decisions, so they should be split into their own issues.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

documentation origin/outsider

Epic: #5171


Problem statement

The Video Model Cost Picker cheapest routes are $0.00 job cost and do not call cloud video APIs:

Route Mechanism Picker constraints
ambient-loop ffmpeg displacement field on plate Loops natively; good for water/haze/cloth; bad for directional leaf rustle
Local composite Cloud gen (any model) + segmentation mask + ffmpeg composite Freezes background/faces when T2V re-poses subjects
LTX Cinemagraph LoRA ComfyUI + LoRA on Apple Silicon One element moves; ~3 min/clip @ 512×704 M1 Max; MASK INPUT

PraisonAI already uses ffmpeg in:

  • src/praisonai-browser/praisonai_browser/video.py — create_video_ffmpeg, FFmpegVideoEncoder
  • praisonai_tools/video/render.py — edit pipeline rendering

No unified agent-invokable tools implement picker route names.


Architecture: workflows vs providers

These are not VideoProvider registry entries (#02). Implement as:

praisonaiagents/tools/video_workflows.py   # @tool functions
# OR
praisonai_tools/video/workflows/
  ambient_loop.py
  composite_freeze.py
  comfyui_ltx.py

YAML agents call tools by name; cost catalog (#08) marks provider: "workflow", job_cost_usd: 0 for ffmpeg-only steps.


Workflow 1: ambient-loop

Input
  • plate_path: PNG/JPG loop-friendly texture (water, haze, cloth)
  • duration_seconds: output length (picker default 5s per clip)
  • fps: default 24 or 30
  • Optional displacement_strength, period_seconds tuning
Processing
  1. Build ffmpeg filter_complex with periodic displacement (implementation from picker methodology — sine/cosine displacement maps).
  2. Output seamless loop MP4 (verify loop point with ffprobe in test).
Tool signature (proposed)
@tool
def video_ambient_loop(
    plate_path: str,
    output_path: str,
    duration_seconds: float = 5.0,
    fps: int = 24,
) -> str:
    """Create seamless ambient loop video from still plate via ffmpeg displacement."""
Failure modes
  • ffmpeg missing on PATH → clear install message (link ffmpeg.org)
  • Non-loop-friendly content → document picker limitation in docstring

Workflow 2: composite-freeze (Local composite)

Input
  • generated_video_path: MP4 from any VideoAgent provider
  • reference_still_path: optional; default first frame of video
  • mask: optional user mask; else auto segment with MediaPipe Selfie Segmentation or person mask model
Processing pipeline
1. Extract reference still (frame 0)
2. Compute mask per frame OR static mask from reference
3. ffmpeg: for each frame, composite still outside mask over generated frame
4. Encode output MP4 (H.264)
Tool signature
@tool
def video_composite_freeze_background(
    generated_video_path: str,
    output_path: str,
    mask_expand_px: int = 4,
    use_mediapipe: bool = True,
) -> str:
Dependencies
  • ffmpeg required
  • mediapipe optional extra: pip install praisonaiagents[video-composite] or document manual install
Picker alignment

Addresses “THE FIX” in artifact: models re-pose people; freezing background via mask is $0 incremental cost in picker table.


Workflow 3: LTX Cinemagraph LoRA (ComfyUI)

Scope

Phase A (required): Documentation + workflow JSON export path:

  • Hardware: Apple M-series recommended (picker)
  • Resolution: 512×704
  • LoRA: LTX Cinemagraph (distilled GGUF Q4 referenced in picker)
  • Expected runtime ~3 min/clip on M1 Max

Phase B (optional): ComfyUI API bridge

@tool
def video_comfyui_ltx_cinemagraph(
    input_image_path: str,
    output_path: str,
    comfyui_base_url: str = "http://127.0.0.1:8188",
    workflow_json_path: str = "...",
) -> str:

Submit prompt via ComfyUI /prompt API; poll /history/{prompt_id}; download output video node.

Do not ship ComfyUI or LoRA weights in PraisonAI wheel.


Agent orchestration pattern

Example team YAML logic (documentation):

  1. If scene tags ∈ {water, haze, cloth} → call video_ambient_loop
  2. Else VideoAgent.generate with cheapest supported cloud route from catalog (#08)
  3. If evaluate_motion_on_faces detects drift → video_composite_freeze_background
  4. If user needs single-element loop → ComfyUI doc path

Cost estimator (#08)

Catalog entries:

  "id": "ambient-loop",
  "pricing": { "mode": "free" },
  "flags": ["FREE", "LOOPS NATIVELY"],
  "supported": true,
  "implementation": "tool:video_ambient_loop"

Composite workflow: cloud_cost + 0 for ffmpeg step.


Testing

Test Marker
ambient-loop 1s output @pytest.mark.requires_ffmpeg
composite with synthetic 10-frame MP4 + fixed mask skip mediapipe in CI
ComfyUI bridge @pytest.mark.integration mock HTTP

Use tiny fixtures in tests/fixtures/video/ (64×64 plate PNG).


Acceptance criteria

  • video_ambient_loop tool + documented CLI equivalent
  • video_composite_freeze_background accepts VideoAgent output path
  • ComfyUI guide in docs/video/LTX_CINEMAGRAPH.md with graph export instructions
  • Matrix: three picker rows Supported (workflow)
  • Agent example in examples/python/video/workflows/

Dependencies

Cloud generation optional for composite step (#02–#06)

Related

Epic #01, #08, #10

References

  • praisonai_tools/video/ — probe, render, motion_graphics
  • praisonai_browser/video.py — ffmpeg helpers

Appendix: ffmpeg filter notes

  • ambient-loop: periodic displacement; verify loop point with ffprobe duration.
  • composite: blend masked regions using alphamerge + overlay chain.
  • Use yuv420p pixel format for compatibility with players.

Appendix: Tool test matrix

Test Marker
test_ambient_1 requires_ffmpeg
test_ambient_2 requires_ffmpeg
test_ambient_3 requires_ffmpeg
test_ambient_4 requires_ffmpeg
test_ambient_5 requires_ffmpeg
test_ambient_6 requires_ffmpeg
test_ambient_7 requires_ffmpeg
test_ambient_8 requires_ffmpeg
test_ambient_9 requires_ffmpeg
test_ambient_10 requires_ffmpeg
test_ambient_11 requires_ffmpeg
test_ambient_12 requires_ffmpeg
test_ambient_13 requires_ffmpeg
test_ambient_14 requires_ffmpeg
test_ambient_15 requires_ffmpeg
test_ambient_16 requires_ffmpeg
test_ambient_17 requires_ffmpeg
test_ambient_18 requires_ffmpeg







ffmpeg CLI examples

ffmpeg -version  # prerequisite
# ambient-loop: see docs/video/workflows/ambient-loop.md
# composite-freeze: see docs/video/workflows/composite-freeze.md

Workflow integration tests (enumeration)

Test id Scenario Fixture Expected
test_workflow_01 Scenario 1 mock HTTP pass
test_workflow_02 Scenario 2 mock HTTP pass
test_workflow_03 Scenario 3 mock HTTP pass
test_workflow_04 Scenario 4 mock HTTP pass
test_workflow_05 Scenario 5 mock HTTP pass
test_workflow_06 Scenario 6 mock HTTP pass
test_workflow_07 Scenario 7 mock HTTP pass
test_workflow_08 Scenario 8 mock HTTP pass
test_workflow_09 Scenario 9 mock HTTP pass
test_workflow_10 Scenario 10 mock HTTP pass
test_workflow_11 Scenario 11 mock HTTP pass
test_workflow_12 Scenario 12 mock HTTP pass
test_workflow_13 Scenario 13 mock HTTP pass
test_workflow_14 Scenario 14 mock HTTP pass
test_workflow_15 Scenario 15 mock HTTP pass
test_workflow_16 Scenario 16 mock HTTP pass
test_workflow_17 Scenario 17 mock HTTP pass
test_workflow_18 Scenario 18 mock HTTP pass
test_workflow_19 Scenario 19 mock HTTP pass
test_workflow_20 Scenario 20 mock HTTP pass
test_workflow_21 Scenario 21 mock HTTP pass
test_workflow_22 Scenario 22 mock HTTP pass
test_workflow_23 Scenario 23 mock HTTP pass
test_workflow_24 Scenario 24 mock HTTP pass
test_workflow_25 Scenario 25 mock HTTP pass
test_workflow_26 Scenario 26 mock HTTP pass
test_workflow_27 Scenario 27 mock HTTP pass
test_workflow_28 Scenario 28 mock HTTP pass

Agent tool contracts

Tool Input Output
video_ambient_loop plate_path, output_path MP4 path
video_composite_freeze_background generated_mp4, output_path MP4 path
video_comfyui_ltx_cinemagraph image, comfy URL MP4 path or doc link
Lingua principale
Python
Stelle
9.3k
Fork
1.5k
Merge medio
1g 5h
PR unite (30g)
520

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di MervinPraison/PraisonAI

Tutte le issue di MervinPraison/PraisonAI

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.