[Feature] Add media generation and reranking resources with DashScope providers
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- backend-api-design
Research direction
Read the existing ChatMessage/provider integration issues #1059 and #1374, then map the Python resource contracts, Java-to-Python resource bridge, and YAML schemas and aliases. Start with the proposed MediaModel and RerankModel contracts and verify how current resources handle typed content and provider calls. Done means DashScope supports the listed capabilities with focused contract and protocol tests, Java wrappers and two runnable Python examples.
Written by the indexing model from the issue text.
Description
Search before asking
- I searched the existing issues. #1059 and #1374 cover multimodal ChatMessage/provider integration; this proposal adds separate media-generation and reranking resources.
Description
Applications need to synthesize speech, transcribe recordings, generate images and videos, and rerank retrieval candidates without treating every operation as a chat completion. Long-running provider jobs also need a serializable handle so callers can query their state and resume waiting without submitting a new job.
Introduce provider-independent MediaModel and RerankModel resource contracts, with DashScope as the first Python provider and Java access through the existing Python resource bridge.
| Resource | Capability | Execution and result |
|---|---|---|
| MediaModel | Speech synthesis | Synchronous request returning an audio URL |
| MediaModel | File transcription | Asynchronous job returning transcript blocks and metadata |
| MediaModel | Image generation/editing | Asynchronous job returning image URLs |
| MediaModel | Video generation | Asynchronous job returning a video URL |
| RerankModel | Text and multimodal reranking | Ranked candidate indices, relevance scores, and optional original content |
Proposed contract
- Reuse the existing typed content blocks. Each media resource binds a model and an explicit task; model names do not select the invocation protocol.
- Expose
generate()for a completed result andsubmit/fetch/wait/cancelfor asynchronous jobs. Synchronous speech synthesis has no job operations. - Keep asynchronous polling, terminal states, and timeout behavior common across task types. A polling timeout leaves the provider task running and retains its handle.
- Let callers persist a successful
submit()outcome and resumewait()separately. This does not guarantee exactly-once task creation if the handle was not durably recorded. - Keep request indices stable in rerank results and preserve the provider's ranking order.
- Support Python decorators, Java annotations, programmatic descriptors, and YAML declarations, with aligned Java/Python results and control-flow exceptions.
Delivery scope and verification
The initial implementation includes Python providers, Java-to-Python wrappers, YAML schemas/aliases, focused contract and protocol tests, and two runnable Python examples. Native Java providers, reverse Python-to-Java wrappers, and built-in request Events/Actions are outside this delivery.
DashScope SDK 2.0.4 or later in the 2.x series is required. Existing Chat and Embedding source APIs remain unchanged; the SDK upgrade needs compatibility and installation checks.
User documentation will follow after the first-phase DashScope multimodal PR #1374 merges, so the combined capability guide describes the final public contracts. The implementation PR will be marked doc-needed.
Are you willing to submit a PR?
- I'm willing to submit a PR.
- Dominant language
- Java
- Stars
- 491
- Forks
- 184
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 69
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/flink-agents
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
apache/flink-agents#1395 ·
Maintainers usually reply within 1 day
-
bug priority/major
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
apache/flink-agents#1394 · 1 comment ·
Maintainers usually reply within 1 day
-
[Bug][plan][java][python] BashTool accepts unlisted bare commands when cwd is allowedPossibly taken @wenjin272 claimed this today. Openbug priority/major
apache/flink-agents#1392 · 1 assignee ·
Maintainers usually reply within 1 day
-
[Tech Debt] Unify embedding model result APIs across Java and PythonPossibly taken @wenjin272 claimed this 4 days ago. Openpriority/major tech debt
apache/flink-agents#1385 · 1 assignee ·
Maintainers usually reply within 1 day
-
[Feature][api] Standardize framework duration configurationPossibly taken @HsbcJone claimed this 4 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 43/100
apache/flink-agents#1384 ·
Maintainers usually reply within 1 day
All issues in apache/flink-agents
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
floci-io/floci#8666 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
quarkiverse/quarkiverse#476 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
chinabugotech/hutool#4364 · 1 comment ·
Maintainers usually reply within 1 day
-
lesson-submission pending-review
Difficulty 1/5 Under an hour Newbie friendliness 72/100
Ikalus1988/MisakaNet#5115 ·
Maintainers usually reply within 1 day
-
bug good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
crowdin/crowdin-api-client-java#441 ·
Maintainers usually reply within 1 day