infinity-provider-rig: "Http client error: error sending request" is classified Fatal, so turns fail instead of retrying (20-32% of turns at 8-128 agents against llama-server)

Open Beginner friendly
  • Is something wrong?
#187 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
84/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
backend, distributed-systems

Research direction

Read crates/infinity-provider-rig/src/convert.rs, focusing on classify_provider_message and error_from_rig; the issue includes a proposed classification change and a unit test. Start by running cargo test -p infinity-provider-rig to check the existing behavior. Done means the SSE send error is classified as transient and the provider crate tests pass.

Written by the indexing model from the issue text.

Description

Summary

With infinity-provider-rig and rig-core 0.31's OpenAI Chat Completions client pointed at llama-server, 20-32% of agent turns fail under concurrency with:

ProviderError: Http client error: error sending request for url (http://127.0.0.1:8098/v1/chat/completions)

The request fails at send time, on a pooled keep-alive socket that the server has already closed. Infinity's completion loop would retry it, but the bridge classifies the error as ErrorClass::Fatal, so the turn ends. Mapping this message to Transient in classify_provider_message fixes it: 0 failed turns, and throughput matches a plain client. A patch and a unit test are below.

Environment
  • infinity at c1a7b05 (current main)
  • rig-core 0.31.0, reqwest 0.13
  • llama-server, llama.cpp build 11753 (d8d92973c), -np 8 -c 65536 --jinja, Qwen2.5-7B-Instruct Q4_K_M, on 127.0.0.1:8098
  • AgentSystemBuilder::new_local with one tool (a calculator), streaming completions, single-threaded tokio runtime with a LocalSet
Reproduce

N agents on one local system, 5 turns each. Each turn asks the model to call the tool and then reply. The client is built the default way:

let client = rig::providers::openai::CompletionsClient::<reqwest::Client>::builder()
    .api_key::<rig::client::BearerAuth>("sk-local")
    .base_url("http://127.0.0.1:8098/v1")
    .build()?;
let model = RigCompletionModel::new(client.completion_model("local"));

At N=1 nothing fails. At N=8 and above, turns start failing with the error above and are not retried.

Results

Same server and workload, all runs in one session. "Python" is a plain asyncio client that opens a new connection per request, included as a reference.

agents as shipped: failed turns as shipped: OK turns/s with patch: failed turns with patch: OK turns/s Python: OK turns/s
8 8 of 40 (20.0%) 4.77 0 5.31 5.32
32 38 of 160 (23.8%) 3.36 0 3.71 3.70
128 205 of 640 (32.0%) 3.27 0 3.39 3.46

With the patch, the stale socket is still hit (7, 65 and 295 retries at N=8, 32 and 128), but every retry succeeds. Each retry waits the 1 s backoff, which shows at low concurrency: at N=8, turn p95 was 2.27 s with the patch against 1.56 s with the no-pool workaround below. At N=32 and above, queueing at the 8 server slots hides it. An earlier run of the unpatched bridge showed 17.5%, 24.4% and 28.1% failed turns at the same N.

Cause
  1. rig's OpenAI streaming path sends the request lazily inside its SSE event source. A send failure comes out of the stream as a string: rig-core-0.31.0/src/providers/openai/completion/streaming.rs line 275, yield Err(CompletionError::ProviderError(error.to_string()));.
  2. So the error never reaches the HttpError(_) | UrlError(_) => Transient arm of error_from_rig in crates/infinity-provider-rig/src/convert.rs. It arrives as ProviderError("Http client error: error sending request ...").
  3. classify_provider_message does not recognise that text and returns ErrorClass::Fatal. The event processor retries mid-stream Transient errors (1 s backoff, up to MAX_COMPLETION_RETRIES = 10) but not Fatal ones, so the turn fails.

The failure rate grows with concurrency because more idle pooled sockets go stale between requests. llama-server closes the connection after a streamed response.

Proposed fix

Treat rig's transport errors as transient in classify_provider_message:

--- a/crates/infinity-provider-rig/src/convert.rs
+++ b/crates/infinity-provider-rig/src/convert.rs
@@ fn classify_provider_message(message: &str) -> proto::ErrorClass {
-    if msg.contains("internal server error") || msg.contains("service unavailable") {
+    // rig 0.31 streams over SSE and turns transport failures (stale pooled socket,
+    // connection reset) into ProviderError(String), so they never reach the
+    // HttpError -> Transient arm of error_from_rig. Retry them here.
+    if msg.contains("internal server error")
+        || msg.contains("service unavailable")
+        || msg.contains("http client error")
+        || msg.contains("error sending request")
+    {
         return proto::ErrorClass::Transient;
     }

plus a test:

#[test]
fn sse_send_error_is_transient() {
    let e = error_from_rig(rig::completion::CompletionError::ProviderError(
        "Http client error: error sending request".to_owned(),
    ));
    assert!(matches!(e.class(), proto::ErrorClass::Transient), "{e:?}");
}

cargo test -p infinity-provider-rig passes (4 unit tests, plus the doc test). Http client error: is the prefix of rig's http_client::Error::Instance, which wraps the underlying reqwest error, so the match is limited to transport failures. Status-code errors use a different message.

Workaround for users

Give rig a client with no idle pool:

let http = reqwest::Client::builder().pool_max_idle_per_host(0).build()?;
let client = rig::providers::openai::CompletionsClient::<reqwest::Client>::builder()
    .api_key::<rig::client::BearerAuth>("sk-local")
    .base_url("http://127.0.0.1:8098/v1")
    .http_client(http)
    .build()?;

This alone also gave 0 failed turns (OK turns/s 5.39, 3.62 and 3.40 at N=8, 32 and 128). It costs a new connection per request, which is free on localhost but means a new TLS handshake against a remote endpoint. That makes it a workaround, not a fix. The classification change covers any OpenAI-compatible server that drops keep-alive connections.

It may also be worth raising with rig that stream send errors lose their HttpError type. I can open a PR with the patch and test above if that would help.

Dominant language
Rust
Stars
716
Forks
81
Avg merge
1d 18h
Merged PRs (30d)
11

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from hydro-project/infinity

All issues in hydro-project/infinity

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.