InferenceClient: provide generator-based bulk inference

Open

#68 opened on Aug 5, 2020 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty: 4/5
Estimated time: 3-5 days
Newbie friendliness: 35/100

Issue type: Feature
Clarity: Mostly clear
Activity status: Stale
Tech stack: python
Domain: api

Research direction

Locate InferenceClient and do_bulk_inference, then review the related issue #62 for context. Determine whether the existing API or a separate method should yield individual chunks; done means callers can handle errors without losing progress and do_bulk_inference uses the generator-based behavior.

Written by the indexing model from the issue text.

Description

The current implementation will try to perform inference for all chunks. If an exception occurs, the progress is lost.

By yielding the individual chunks, the caller can be in charge of error handling.

We can either change the existing API or introduce a separate method (and implement do_bulk_inference using the new method).

Contributor guide

Open the contributing guide

First steps

Read the whole issue, then the project's contributing guide.
Comment on the issue to say you are picking it up — it saves two people doing the same work.
Fork the repository and make your change on a branch.
Open a pull request that references the issue number.

Similar issues

`Array.initialize` and `Array.repeat` throw on a negative count Open

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

gren-lang/core#167 · Opened Sep 25, 2026
fix(pg-meta): schemas.update and schemas.remove DO blocks fail when a name contains "$$" Open

external-issue to-triage

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

supabase/supabase#50882 · Opened Sep 24, 2026
wkt: `WKTFieldMask.paths` is an immutable `public let paths: [String]` with no default `init(paths: [String] = [])` or `.with(_:)` helper, preventing in-place mutation Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

googleapis/google-cloud-swift#1151 · Opened Sep 24, 2026
SqliteSaver.list(filter=...) silently misses nested metadata values containing non-ASCII text Open

external

Difficulty 2/5 1-3 hours Newbie friendliness 65/100

langchain-ai/langgraph#9074 · 1 comment · Opened Sep 24, 2026
Correct FileCitationBody.index documentation to describe its position in output text Open

documentation specification

Difficulty 1/5 Under an hour Newbie friendliness 90/100

openai/openai-openapi#584 · Opened Sep 24, 2026

More Backend & API Design issues