[video] Zero-cost ffmpeg and ComfyUI cinemagraph workflows
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
調査の方向性
Start with the existing ffmpeg helpers in src/praisonai-browser/praisonai_browser/video.py (create_video_ffmpeg, FFmpegVideoEncoder) and the render pipeline in praisonai_tools/video/render.py. Pick one workflow, such as video_ambient_loop, and build it as a single @tool function, checking the output loop point with ffprobe. Done means a tiny 64×64 fixture plate produces a seamless MP4 in a test marked requires_ffmpeg. The composite and ComfyUI workflows depend on separate design decisions, so they should be split into their own issues.
索引モデルが issue の本文から書いたものです。
説明
Epic: #5171
Problem statement
The Video Model Cost Picker cheapest routes are $0.00 job cost and do not call cloud video APIs:
| Route | Mechanism | Picker constraints |
|---|---|---|
| ambient-loop | ffmpeg displacement field on plate | Loops natively; good for water/haze/cloth; bad for directional leaf rustle |
| Local composite | Cloud gen (any model) + segmentation mask + ffmpeg composite | Freezes background/faces when T2V re-poses subjects |
| LTX Cinemagraph LoRA | ComfyUI + LoRA on Apple Silicon | One element moves; ~3 min/clip @ 512×704 M1 Max; MASK INPUT |
PraisonAI already uses ffmpeg in:
src/praisonai-browser/praisonai_browser/video.py—create_video_ffmpeg,FFmpegVideoEncoderpraisonai_tools/video/render.py— edit pipeline rendering
No unified agent-invokable tools implement picker route names.
Architecture: workflows vs providers
These are not VideoProvider registry entries (#02). Implement as:
praisonaiagents/tools/video_workflows.py # @tool functions
# OR
praisonai_tools/video/workflows/
ambient_loop.py
composite_freeze.py
comfyui_ltx.py
YAML agents call tools by name; cost catalog (#08) marks provider: "workflow", job_cost_usd: 0 for ffmpeg-only steps.
Workflow 1: ambient-loop
Input
plate_path: PNG/JPG loop-friendly texture (water, haze, cloth)duration_seconds: output length (picker default 5s per clip)fps: default 24 or 30- Optional
displacement_strength,period_secondstuning
Processing
- Build ffmpeg
filter_complexwith periodic displacement (implementation from picker methodology — sine/cosine displacement maps). - Output seamless loop MP4 (verify loop point with
ffprobein test).
Tool signature (proposed)
@tool
def video_ambient_loop(
plate_path: str,
output_path: str,
duration_seconds: float = 5.0,
fps: int = 24,
) -> str:
"""Create seamless ambient loop video from still plate via ffmpeg displacement."""
Failure modes
- ffmpeg missing on PATH → clear install message (link ffmpeg.org)
- Non-loop-friendly content → document picker limitation in docstring
Workflow 2: composite-freeze (Local composite)
Input
generated_video_path: MP4 from anyVideoAgentproviderreference_still_path: optional; default first frame of videomask: optional user mask; else auto segment with MediaPipe Selfie Segmentation or person mask model
Processing pipeline
1. Extract reference still (frame 0)
2. Compute mask per frame OR static mask from reference
3. ffmpeg: for each frame, composite still outside mask over generated frame
4. Encode output MP4 (H.264)
Tool signature
@tool
def video_composite_freeze_background(
generated_video_path: str,
output_path: str,
mask_expand_px: int = 4,
use_mediapipe: bool = True,
) -> str:
Dependencies
ffmpegrequiredmediapipeoptional extra:pip install praisonaiagents[video-composite]or document manual install
Picker alignment
Addresses “THE FIX” in artifact: models re-pose people; freezing background via mask is $0 incremental cost in picker table.
Workflow 3: LTX Cinemagraph LoRA (ComfyUI)
Scope
Phase A (required): Documentation + workflow JSON export path:
- Hardware: Apple M-series recommended (picker)
- Resolution: 512×704
- LoRA: LTX Cinemagraph (distilled GGUF Q4 referenced in picker)
- Expected runtime ~3 min/clip on M1 Max
Phase B (optional): ComfyUI API bridge
@tool
def video_comfyui_ltx_cinemagraph(
input_image_path: str,
output_path: str,
comfyui_base_url: str = "http://127.0.0.1:8188",
workflow_json_path: str = "...",
) -> str:
Submit prompt via ComfyUI /prompt API; poll /history/{prompt_id}; download output video node.
Do not ship ComfyUI or LoRA weights in PraisonAI wheel.
Agent orchestration pattern
Example team YAML logic (documentation):
- If scene tags ∈
{water, haze, cloth}→ callvideo_ambient_loop - Else
VideoAgent.generatewith cheapest supported cloud route from catalog (#08) - If
evaluate_motion_on_facesdetects drift →video_composite_freeze_background - If user needs single-element loop → ComfyUI doc path
Cost estimator (#08)
Catalog entries:
"id": "ambient-loop",
"pricing": { "mode": "free" },
"flags": ["FREE", "LOOPS NATIVELY"],
"supported": true,
"implementation": "tool:video_ambient_loop"
Composite workflow: cloud_cost + 0 for ffmpeg step.
Testing
| Test | Marker |
|---|---|
| ambient-loop 1s output | @pytest.mark.requires_ffmpeg |
| composite with synthetic 10-frame MP4 + fixed mask | skip mediapipe in CI |
| ComfyUI bridge | @pytest.mark.integration mock HTTP |
Use tiny fixtures in tests/fixtures/video/ (64×64 plate PNG).
Acceptance criteria
-
video_ambient_looptool + documented CLI equivalent -
video_composite_freeze_backgroundacceptsVideoAgentoutput path - ComfyUI guide in
docs/video/LTX_CINEMAGRAPH.mdwith graph export instructions - Matrix: three picker rows Supported (workflow)
- Agent example in
examples/python/video/workflows/
Dependencies
Cloud generation optional for composite step (#02–#06)
Related
Epic #01, #08, #10
References
praisonai_tools/video/— probe, render, motion_graphicspraisonai_browser/video.py— ffmpeg helpers
Appendix: ffmpeg filter notes
- ambient-loop: periodic displacement; verify loop point with ffprobe
duration. - composite: blend masked regions using
alphamerge+overlaychain. - Use yuv420p pixel format for compatibility with players.
Appendix: Tool test matrix
| Test | Marker |
|---|---|
| test_ambient_1 | requires_ffmpeg |
| test_ambient_2 | requires_ffmpeg |
| test_ambient_3 | requires_ffmpeg |
| test_ambient_4 | requires_ffmpeg |
| test_ambient_5 | requires_ffmpeg |
| test_ambient_6 | requires_ffmpeg |
| test_ambient_7 | requires_ffmpeg |
| test_ambient_8 | requires_ffmpeg |
| test_ambient_9 | requires_ffmpeg |
| test_ambient_10 | requires_ffmpeg |
| test_ambient_11 | requires_ffmpeg |
| test_ambient_12 | requires_ffmpeg |
| test_ambient_13 | requires_ffmpeg |
| test_ambient_14 | requires_ffmpeg |
| test_ambient_15 | requires_ffmpeg |
| test_ambient_16 | requires_ffmpeg |
| test_ambient_17 | requires_ffmpeg |
| test_ambient_18 | requires_ffmpeg |
ffmpeg CLI examples
ffmpeg -version # prerequisite
# ambient-loop: see docs/video/workflows/ambient-loop.md
# composite-freeze: see docs/video/workflows/composite-freeze.md
Workflow integration tests (enumeration)
| Test id | Scenario | Fixture | Expected |
|---|---|---|---|
test_workflow_01 |
Scenario 1 | mock HTTP | pass |
test_workflow_02 |
Scenario 2 | mock HTTP | pass |
test_workflow_03 |
Scenario 3 | mock HTTP | pass |
test_workflow_04 |
Scenario 4 | mock HTTP | pass |
test_workflow_05 |
Scenario 5 | mock HTTP | pass |
test_workflow_06 |
Scenario 6 | mock HTTP | pass |
test_workflow_07 |
Scenario 7 | mock HTTP | pass |
test_workflow_08 |
Scenario 8 | mock HTTP | pass |
test_workflow_09 |
Scenario 9 | mock HTTP | pass |
test_workflow_10 |
Scenario 10 | mock HTTP | pass |
test_workflow_11 |
Scenario 11 | mock HTTP | pass |
test_workflow_12 |
Scenario 12 | mock HTTP | pass |
test_workflow_13 |
Scenario 13 | mock HTTP | pass |
test_workflow_14 |
Scenario 14 | mock HTTP | pass |
test_workflow_15 |
Scenario 15 | mock HTTP | pass |
test_workflow_16 |
Scenario 16 | mock HTTP | pass |
test_workflow_17 |
Scenario 17 | mock HTTP | pass |
test_workflow_18 |
Scenario 18 | mock HTTP | pass |
test_workflow_19 |
Scenario 19 | mock HTTP | pass |
test_workflow_20 |
Scenario 20 | mock HTTP | pass |
test_workflow_21 |
Scenario 21 | mock HTTP | pass |
test_workflow_22 |
Scenario 22 | mock HTTP | pass |
test_workflow_23 |
Scenario 23 | mock HTTP | pass |
test_workflow_24 |
Scenario 24 | mock HTTP | pass |
test_workflow_25 |
Scenario 25 | mock HTTP | pass |
test_workflow_26 |
Scenario 26 | mock HTTP | pass |
test_workflow_27 |
Scenario 27 | mock HTTP | pass |
test_workflow_28 |
Scenario 28 | mock HTTP | pass |
Agent tool contracts
| Tool | Input | Output |
|---|---|---|
| video_ambient_loop | plate_path, output_path | MP4 path |
| video_composite_freeze_background | generated_mp4, output_path | MP4 path |
| video_comfyui_ltx_cinemagraph | image, comfy URL | MP4 path or doc link |
- 主要言語
- Python
- スター
- 9.3k
- フォーク
- 1.5k
- 平均マージ
- 1日 5時間
- マージ済み PR(30日)
- 520
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
MervinPraison/PraisonAI のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
MervinPraison/PraisonAI#6710 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
claude
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
MervinPraison/PraisonAI#6707 ·
メンテナーはふだん 1 日以内に返信
-
bug claude
難易度 1/5 1時間未満 初心者へのやさしさ 90/100
MervinPraison/PraisonAI#6671 ·
メンテナーはふだん 1 日以内に返信
-
Compaction budget regression fixture remains below budget with accurate tokenization再び着手できるかも このイシューのプルリクエストはマージされずにクローズされました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
MervinPraison/PraisonAI#5418 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
-
bug claude documentation
難易度 2/5 1〜3時間 初心者へのやさしさ 70/100
MervinPraison/PraisonAI#5341 · コメント 3 件 ·
メンテナーはふだん 1 日以内に返信
MervinPraison/PraisonAI の issue をすべて見る
似ている issue
-
cognition
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 78/100
collabnix/awesome-mcp-lists#433 ·
メンテナーはふだん 4 日以内に返信
-
CrossEntropyFilter raises on zero probabilities and single-class results対応中かも @NopeCharon が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
datajoint/datajoint-python#1703 ·
メンテナーはふだん 6 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
Archmonger/ServeStatic#129 ·