[video] Zero-cost ffmpeg and ComfyUI cinemagraph workflows
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 22/100
Direzione di ricerca
Start with the existing ffmpeg helpers in src/praisonai-browser/praisonai_browser/video.py (create_video_ffmpeg, FFmpegVideoEncoder) and the render pipeline in praisonai_tools/video/render.py. Pick one workflow, such as video_ambient_loop, and build it as a single @tool function, checking the output loop point with ffprobe. Done means a tiny 64×64 fixture plate produces a seamless MP4 in a test marked requires_ffmpeg. The composite and ComfyUI workflows depend on separate design decisions, so they should be split into their own issues.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Epic: #5171
Problem statement
The Video Model Cost Picker cheapest routes are $0.00 job cost and do not call cloud video APIs:
| Route | Mechanism | Picker constraints |
|---|---|---|
| ambient-loop | ffmpeg displacement field on plate | Loops natively; good for water/haze/cloth; bad for directional leaf rustle |
| Local composite | Cloud gen (any model) + segmentation mask + ffmpeg composite | Freezes background/faces when T2V re-poses subjects |
| LTX Cinemagraph LoRA | ComfyUI + LoRA on Apple Silicon | One element moves; ~3 min/clip @ 512×704 M1 Max; MASK INPUT |
PraisonAI already uses ffmpeg in:
src/praisonai-browser/praisonai_browser/video.py—create_video_ffmpeg,FFmpegVideoEncoderpraisonai_tools/video/render.py— edit pipeline rendering
No unified agent-invokable tools implement picker route names.
Architecture: workflows vs providers
These are not VideoProvider registry entries (#02). Implement as:
praisonaiagents/tools/video_workflows.py # @tool functions
# OR
praisonai_tools/video/workflows/
ambient_loop.py
composite_freeze.py
comfyui_ltx.py
YAML agents call tools by name; cost catalog (#08) marks provider: "workflow", job_cost_usd: 0 for ffmpeg-only steps.
Workflow 1: ambient-loop
Input
plate_path: PNG/JPG loop-friendly texture (water, haze, cloth)duration_seconds: output length (picker default 5s per clip)fps: default 24 or 30- Optional
displacement_strength,period_secondstuning
Processing
- Build ffmpeg
filter_complexwith periodic displacement (implementation from picker methodology — sine/cosine displacement maps). - Output seamless loop MP4 (verify loop point with
ffprobein test).
Tool signature (proposed)
@tool
def video_ambient_loop(
plate_path: str,
output_path: str,
duration_seconds: float = 5.0,
fps: int = 24,
) -> str:
"""Create seamless ambient loop video from still plate via ffmpeg displacement."""
Failure modes
- ffmpeg missing on PATH → clear install message (link ffmpeg.org)
- Non-loop-friendly content → document picker limitation in docstring
Workflow 2: composite-freeze (Local composite)
Input
generated_video_path: MP4 from anyVideoAgentproviderreference_still_path: optional; default first frame of videomask: optional user mask; else auto segment with MediaPipe Selfie Segmentation or person mask model
Processing pipeline
1. Extract reference still (frame 0)
2. Compute mask per frame OR static mask from reference
3. ffmpeg: for each frame, composite still outside mask over generated frame
4. Encode output MP4 (H.264)
Tool signature
@tool
def video_composite_freeze_background(
generated_video_path: str,
output_path: str,
mask_expand_px: int = 4,
use_mediapipe: bool = True,
) -> str:
Dependencies
ffmpegrequiredmediapipeoptional extra:pip install praisonaiagents[video-composite]or document manual install
Picker alignment
Addresses “THE FIX” in artifact: models re-pose people; freezing background via mask is $0 incremental cost in picker table.
Workflow 3: LTX Cinemagraph LoRA (ComfyUI)
Scope
Phase A (required): Documentation + workflow JSON export path:
- Hardware: Apple M-series recommended (picker)
- Resolution: 512×704
- LoRA: LTX Cinemagraph (distilled GGUF Q4 referenced in picker)
- Expected runtime ~3 min/clip on M1 Max
Phase B (optional): ComfyUI API bridge
@tool
def video_comfyui_ltx_cinemagraph(
input_image_path: str,
output_path: str,
comfyui_base_url: str = "http://127.0.0.1:8188",
workflow_json_path: str = "...",
) -> str:
Submit prompt via ComfyUI /prompt API; poll /history/{prompt_id}; download output video node.
Do not ship ComfyUI or LoRA weights in PraisonAI wheel.
Agent orchestration pattern
Example team YAML logic (documentation):
- If scene tags ∈
{water, haze, cloth}→ callvideo_ambient_loop - Else
VideoAgent.generatewith cheapest supported cloud route from catalog (#08) - If
evaluate_motion_on_facesdetects drift →video_composite_freeze_background - If user needs single-element loop → ComfyUI doc path
Cost estimator (#08)
Catalog entries:
"id": "ambient-loop",
"pricing": { "mode": "free" },
"flags": ["FREE", "LOOPS NATIVELY"],
"supported": true,
"implementation": "tool:video_ambient_loop"
Composite workflow: cloud_cost + 0 for ffmpeg step.
Testing
| Test | Marker |
|---|---|
| ambient-loop 1s output | @pytest.mark.requires_ffmpeg |
| composite with synthetic 10-frame MP4 + fixed mask | skip mediapipe in CI |
| ComfyUI bridge | @pytest.mark.integration mock HTTP |
Use tiny fixtures in tests/fixtures/video/ (64×64 plate PNG).
Acceptance criteria
-
video_ambient_looptool + documented CLI equivalent -
video_composite_freeze_backgroundacceptsVideoAgentoutput path - ComfyUI guide in
docs/video/LTX_CINEMAGRAPH.mdwith graph export instructions - Matrix: three picker rows Supported (workflow)
- Agent example in
examples/python/video/workflows/
Dependencies
Cloud generation optional for composite step (#02–#06)
Related
Epic #01, #08, #10
References
praisonai_tools/video/— probe, render, motion_graphicspraisonai_browser/video.py— ffmpeg helpers
Appendix: ffmpeg filter notes
- ambient-loop: periodic displacement; verify loop point with ffprobe
duration. - composite: blend masked regions using
alphamerge+overlaychain. - Use yuv420p pixel format for compatibility with players.
Appendix: Tool test matrix
| Test | Marker |
|---|---|
| test_ambient_1 | requires_ffmpeg |
| test_ambient_2 | requires_ffmpeg |
| test_ambient_3 | requires_ffmpeg |
| test_ambient_4 | requires_ffmpeg |
| test_ambient_5 | requires_ffmpeg |
| test_ambient_6 | requires_ffmpeg |
| test_ambient_7 | requires_ffmpeg |
| test_ambient_8 | requires_ffmpeg |
| test_ambient_9 | requires_ffmpeg |
| test_ambient_10 | requires_ffmpeg |
| test_ambient_11 | requires_ffmpeg |
| test_ambient_12 | requires_ffmpeg |
| test_ambient_13 | requires_ffmpeg |
| test_ambient_14 | requires_ffmpeg |
| test_ambient_15 | requires_ffmpeg |
| test_ambient_16 | requires_ffmpeg |
| test_ambient_17 | requires_ffmpeg |
| test_ambient_18 | requires_ffmpeg |
ffmpeg CLI examples
ffmpeg -version # prerequisite
# ambient-loop: see docs/video/workflows/ambient-loop.md
# composite-freeze: see docs/video/workflows/composite-freeze.md
Workflow integration tests (enumeration)
| Test id | Scenario | Fixture | Expected |
|---|---|---|---|
test_workflow_01 |
Scenario 1 | mock HTTP | pass |
test_workflow_02 |
Scenario 2 | mock HTTP | pass |
test_workflow_03 |
Scenario 3 | mock HTTP | pass |
test_workflow_04 |
Scenario 4 | mock HTTP | pass |
test_workflow_05 |
Scenario 5 | mock HTTP | pass |
test_workflow_06 |
Scenario 6 | mock HTTP | pass |
test_workflow_07 |
Scenario 7 | mock HTTP | pass |
test_workflow_08 |
Scenario 8 | mock HTTP | pass |
test_workflow_09 |
Scenario 9 | mock HTTP | pass |
test_workflow_10 |
Scenario 10 | mock HTTP | pass |
test_workflow_11 |
Scenario 11 | mock HTTP | pass |
test_workflow_12 |
Scenario 12 | mock HTTP | pass |
test_workflow_13 |
Scenario 13 | mock HTTP | pass |
test_workflow_14 |
Scenario 14 | mock HTTP | pass |
test_workflow_15 |
Scenario 15 | mock HTTP | pass |
test_workflow_16 |
Scenario 16 | mock HTTP | pass |
test_workflow_17 |
Scenario 17 | mock HTTP | pass |
test_workflow_18 |
Scenario 18 | mock HTTP | pass |
test_workflow_19 |
Scenario 19 | mock HTTP | pass |
test_workflow_20 |
Scenario 20 | mock HTTP | pass |
test_workflow_21 |
Scenario 21 | mock HTTP | pass |
test_workflow_22 |
Scenario 22 | mock HTTP | pass |
test_workflow_23 |
Scenario 23 | mock HTTP | pass |
test_workflow_24 |
Scenario 24 | mock HTTP | pass |
test_workflow_25 |
Scenario 25 | mock HTTP | pass |
test_workflow_26 |
Scenario 26 | mock HTTP | pass |
test_workflow_27 |
Scenario 27 | mock HTTP | pass |
test_workflow_28 |
Scenario 28 | mock HTTP | pass |
Agent tool contracts
| Tool | Input | Output |
|---|---|---|
| video_ambient_loop | plate_path, output_path | MP4 path |
| video_composite_freeze_background | generated_mp4, output_path | MP4 path |
| video_comfyui_ltx_cinemagraph | image, comfy URL | MP4 path or doc link |
- Lingua principale
- Python
- Stelle
- 9.3k
- Fork
- 1.5k
- Merge medio
- 1g 5h
- PR unite (30g)
- 520
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di MervinPraison/PraisonAI
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
MervinPraison/PraisonAI#6710 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
claude
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
MervinPraison/PraisonAI#6707 ·
I maintainer di solito rispondono entro 1 giorno
-
bug claude
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 90/100
MervinPraison/PraisonAI#6671 ·
I maintainer di solito rispondono entro 1 giorno
-
Compaction budget regression fixture remains below budget with accurate tokenizationForse di nuovo libera Una pull request per questa issue è stata chiusa senza essere unita. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
MervinPraison/PraisonAI#5418 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
bug claude documentation
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
MervinPraison/PraisonAI#5341 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di MervinPraison/PraisonAI
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
I maintainer di solito rispondono entro 2 giorni
-
[Bug]: LANGSMITH_SAMPLING_RATE ignores fractional values (isdigit check), traces 100% of requestsAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
I maintainer di solito rispondono entro 1 giorno
-
platform/backend tests
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
Significant-Gravitas/AutoGPT#17100 ·
I maintainer di solito rispondono entro 1 giorno
-
bug good first issue
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 61/100
scverse/infercnvpy#238 ·
I maintainer di solito rispondono entro 1 giorno