[video] Replicate video models (Predictions API)

Aperta
#5,174 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
12/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
ai, api

Direzione di ricerca

This is an epic-level design task spanning many model routes, so it is not a first issue. Start by reading examples/python/providers/replicate/replicate_example.py and the proposed praisonaiagents/video/adapters/replicate.py. The acceptance criteria section is empty and the integration test table is placeholder rows, so 'done' is not yet defined. Wait for a maintainer to split this into smaller route-specific issues.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug claude documentation

Epic: #5171


Problem statement

Replicate hosts many Video Model Cost Picker routes. Each model is a versioned prediction with a unique OpenAPI input schema. PraisonAI only documents Replicate for text LLMs (examples/python/providers/replicate/replicate_example.py → llm="replicate/meta/llama-3.1-8b-instruct").

There is no ReplicateVideoProvider. Desktop product builds may hardcode MiniMax video-01 on Replicate; that slug does not appear in the open praisonai-desktop engine source.


Replicate HTTP API (implementation reference)

Base: https://api.replicate.com/v1

Authentication:

Authorization: Bearer $REPLICATE_API_TOKEN
Create prediction
POST /predictions
Content-Type: application/json
{
  "version": "<model_version_hash_or_owner/name:version>",
  "input": {
    "prompt": "...",
    "duration": 5
  }
}

Alternative: "model": "owner/name:version" form per Replicate docs evolution — implement using official replicate Python package if already a transitive dependency, else raw REST.

Response:

{
  "id": "uuid",
  "status": "starting",
  "urls": { "get": "https://api.replicate.com/v1/predictions/..." },
  "output": null
}

Statuses: starting, processing, succeeded, failed, canceled.

Docs: https://replicate.com/docs/reference/http

Poll prediction
GET /predictions/{id}

When status == "succeeded", output may be:

  • String URL to MP4
  • Array of URLs (take index 0 or join per model docs)
  • Object with nested video field — must be schema-driven
Download video

HTTP GET on output URL (often https://replicate.delivery/...). Not authenticated. Stream to bytes for VideoAgent.content().


PraisonAI model string format

replicate/<owner>/<model_name>
replicate/<owner>/<model_name>:<version_id>

Registry (#02) strips prefix and resolves version:

  1. If version pinned in string, use it.
  2. Else lookup video_routes_catalog.json (#08) for pinned version hash.
  3. Else call Replicate models API for latest version (cache).

Picker routes — input schemas (engineering checklist)

Implement route profiles (YAML or JSON) per model:

p-video (draft)
Field Value
Picker cost ~$0.01/s @ 1080p with draft: true
Min billable 1 second (picker differentiator)
Required kwargs draft=True
agent.generate(
    "Scene description",
    seconds="2",
    draft=True,
)
p-video (standard)
  • draft=False or omit
  • ~$0.04/s @ 1080p
Wan 2.2 i2v Fast
  • Flat $0.05 per run @ 480p (picker)
  • Pricing model in estimator (#08): pricing.mode = "flat_per_run" not $/s
Seedance 1.5 Pro
  • Critical: generate_audio=False in input JSON
  • Picker: audio doubles bill
Luma Ray 3.2
  • loop=True for native loop
  • Mutually exclusive with end_image (picker note) — validate in provider before POST
Kling 2.1
  • end_image requires mode="pro" at 1080p (~$0.09/s)
Wan VACE 14B / inpaint
  • src_mask accepts image or video URI
  • Document ~$0.66/run, ~8 min latency
  • Warn: untouched pixels altered by VAE (picker measurement)
MiniMax video-01
  • Desktop preset; pin exact owner/name:version in catalog
Runway Gen-4.5 (Replicate listing)
  • Distinct from runwayml/gen4_turbo LiteLLM path — separate catalog entry

ReplicateVideoProvider design

File: praisonaiagents/video/adapters/replicate.py

class ReplicateVideoProvider(VideoProvider):
    provider_id = "replicate"

    def generate(self, prompt, *, model, seconds, size, input_reference, **kwargs):
        version, input_dict = self._build_input(model, prompt, seconds, size, input_reference, kwargs)
        prediction = self._create_prediction(version, input_dict)
        return VideoJob(
            id=f"replicate:{prediction['id']}",
            status=self._map_status(prediction['status']),
            model=model,
            provider="replicate",
            raw=prediction,
        )

    def get_status(self, job_id, ...):
        ...

    def download(self, job_id, ...):
        # GET prediction -> extract output URL -> download bytes
_build_input responsibilities
  1. Load route profile for model slug.
  2. Map generic params:
    • prompt → model-specific key (some use caption)
    • seconds → duration, num_frames, or length per schema
    • size → resolution, width/height, or enum
    • input_reference → file URL, base64, or Replicate file upload (implement upload helper if needed)
  3. Merge kwargs last (user overrides).
  4. Run validators (loop vs end_image, audio flag).
File upload path

If model requires image file:

  1. Optional helper replicate_upload_file(path) -> url using Replicate files API.
  2. Document in examples for I2V routes.

Integration surfaces

Surface Change
VideoAgent Via registry
CLI praisonai videos "..." --model replicate/owner/name
MCP model parameter on praisonai.videos.generate
Desktop #09 Replicate group in picker

Testing

Fixtures: tests/fixtures/replicate/prediction_{starting,succeeded,failed}.json

Test Assert
Seedance default example input generate_audio is False
Luma loop + end_image ValidationError before HTTP
Wan 2.2 pricing hook catalog flat_run flag
succeeded output URL array download called with first URL

Examples (required)

File Route
video_replicate_pvideo_draft.py p-video draft
video_replicate_wan_i2v_fast.py Wan 2.2 i2v
video_replicate_luma_loop.py Luma Ray 3.2 loop
video_replicate_seedance_15.py Seedance 1.5 Pro silent
video_replicate_minimax_video01.py Desktop parity

Acceptance criteria








Replicate input field mapping (generic)

VideoAgent Typical Replicate key
prompt prompt / caption
seconds duration / num_frames
size resolution / width+height
input_reference image / first_frame
last_frame end_image / last_frame
draft draft
loop loop
generate_audio generate_audio

Replicate integration tests (enumeration)

Test id Scenario Fixture Expected
test_replicate_01 Scenario 1 mock HTTP pass
test_replicate_02 Scenario 2 mock HTTP pass
test_replicate_03 Scenario 3 mock HTTP pass
test_replicate_04 Scenario 4 mock HTTP pass
test_replicate_05 Scenario 5 mock HTTP pass
test_replicate_06 Scenario 6 mock HTTP pass
test_replicate_07 Scenario 7 mock HTTP pass
test_replicate_08 Scenario 8 mock HTTP pass
test_replicate_09 Scenario 9 mock HTTP pass
test_replicate_10 Scenario 10 mock HTTP pass
test_replicate_11 Scenario 11 mock HTTP pass
test_replicate_12 Scenario 12 mock HTTP pass
test_replicate_13 Scenario 13 mock HTTP pass
test_replicate_14 Scenario 14 mock HTTP pass
test_replicate_15 Scenario 15 mock HTTP pass
test_replicate_16 Scenario 16 mock HTTP pass
test_replicate_17 Scenario 17 mock HTTP pass
test_replicate_18 Scenario 18 mock HTTP pass
test_replicate_19 Scenario 19 mock HTTP pass
test_replicate_20 Scenario 20 mock HTTP pass
test_replicate_21 Scenario 21 mock HTTP pass
test_replicate_22 Scenario 22 mock HTTP pass
test_replicate_23 Scenario 23 mock HTTP pass
test_replicate_24 Scenario 24 mock HTTP pass
test_replicate_25 Scenario 25 mock HTTP pass
test_replicate_26 Scenario 26 mock HTTP pass

Profile YAML validation

  • JSON Schema for replicate profile documents in CI
  • Each profile must define: version, input_defaults, validators
  • Version hash pinned per model slug in catalog JSON
  • Profile loader resolves slug → owner/model:version for predictions API
Lingua principale
Python
Stelle
9.3k
Fork
1.5k
Merge medio
1g 5h
PR unite (30g)
520

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di MervinPraison/PraisonAI

Tutte le issue di MervinPraison/PraisonAI

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.