[Feature] Add media generation and reranking resources with DashScope providers

Open
#1,381 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
java, python, yaml

Research direction

Read the existing ChatMessage/provider integration issues #1059 and #1374, then map the Python resource contracts, Java-to-Python resource bridge, and YAML schemas and aliases. Start with the proposed MediaModel and RerankModel contracts and verify how current resources handle typed content and provider calls. Done means DashScope supports the listed capabilities with focused contract and protocol tests, Java wrappers and two runnable Python examples.

Written by the indexing model from the issue text.

Description

Search before asking
  • I searched the existing issues. #1059 and #1374 cover multimodal ChatMessage/provider integration; this proposal adds separate media-generation and reranking resources.
Description

Applications need to synthesize speech, transcribe recordings, generate images and videos, and rerank retrieval candidates without treating every operation as a chat completion. Long-running provider jobs also need a serializable handle so callers can query their state and resume waiting without submitting a new job.

Introduce provider-independent MediaModel and RerankModel resource contracts, with DashScope as the first Python provider and Java access through the existing Python resource bridge.

Resource Capability Execution and result
MediaModel Speech synthesis Synchronous request returning an audio URL
MediaModel File transcription Asynchronous job returning transcript blocks and metadata
MediaModel Image generation/editing Asynchronous job returning image URLs
MediaModel Video generation Asynchronous job returning a video URL
RerankModel Text and multimodal reranking Ranked candidate indices, relevance scores, and optional original content
Proposed contract
  • Reuse the existing typed content blocks. Each media resource binds a model and an explicit task; model names do not select the invocation protocol.
  • Expose generate() for a completed result and submit/fetch/wait/cancel for asynchronous jobs. Synchronous speech synthesis has no job operations.
  • Keep asynchronous polling, terminal states, and timeout behavior common across task types. A polling timeout leaves the provider task running and retains its handle.
  • Let callers persist a successful submit() outcome and resume wait() separately. This does not guarantee exactly-once task creation if the handle was not durably recorded.
  • Keep request indices stable in rerank results and preserve the provider's ranking order.
  • Support Python decorators, Java annotations, programmatic descriptors, and YAML declarations, with aligned Java/Python results and control-flow exceptions.
Delivery scope and verification

The initial implementation includes Python providers, Java-to-Python wrappers, YAML schemas/aliases, focused contract and protocol tests, and two runnable Python examples. Native Java providers, reverse Python-to-Java wrappers, and built-in request Events/Actions are outside this delivery.

DashScope SDK 2.0.4 or later in the 2.x series is required. Existing Chat and Embedding source APIs remain unchanged; the SDK upgrade needs compatibility and installation checks.

User documentation will follow after the first-phase DashScope multimodal PR #1374 merges, so the combined capability guide describes the final public contracts. The implementation PR will be marked doc-needed.

Are you willing to submit a PR?
  • I'm willing to submit a PR.
Dominant language
Java
Stars
491
Forks
184
Avg merge
1d 17h
Merged PRs (30d)
69

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/flink-agents

All issues in apache/flink-agents

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.