Connection refused to the in-cluster Ollama service (ollama:11434) from the memex portal — Standard-Retry's third attempt failed on the unnamed default HttpClient

Open
  • Is something wrong?
#7,678 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
csharp, kubernetes, ollama
Domain
infrastructure, observability-sre

Research direction

Start by checking the ollama pod's restart events and readiness-probe history for the 2026-10-30 10:51 UTC sighting, then compare the deployment's ready replica count and the next watcher window. Read the memex portal logs around that instant to identify the caller behind the unnamed HttpClient. Done means establishing whether this was a restart/readiness gap or a recurring service issue and documenting what the evidence shows; the issue does not specify a code change.

Written by the indexing model from the issue text.

Description

bug sev:M

Observed

A Polly Execution attempt (fail level) on the memex portal records the standard resilience pipeline's third attempt failing with System.Net.Http.HttpRequestException: Connection refused (ollama:11434) → System.Net.Sockets.SocketException (111): Connection refused. The pipeline instance is -standard/(unnamed){path} — the HttpClient factory's unnamed default client, per the naming scheme in {path}. The captured stack is truncated inside System.Net.Http frames, so the portal-side caller below the pipeline is not identified by this sample.

The target ollama:11434 is the memex instance's in-cluster local model service ({path}, chart release {path}): qwen3.6 for the utility tier (thread names, node {path}, classification) and image reading, translation, summaries; bge-m3 for embeddings. A connection refused at that address means the ClusterIP Service had no ready backend listening — it is not an application error from Ollama itself.

Where and when

One line, one pod, one instant: a single failed client request (the third retry attempt), not a sustained outage in what the watcher captured. The line does not line up with the known ollama outage — {path} (0/1 ready, "Down — no replicas ready", severity Critical) was open 04:27–04:32 UTC the same morning and is marked Resolved, hours before this sighting. No chart apply or portal roll is recorded around the sighting (the ChartRelease record was last modified the previous day). Handled: 'True' says the resilience pipeline classified the outcome as {path}

Impact

Whatever job issued the request failed its retry budget. Consumers of this service on memex are {path} AI jobs and embeddings; one refused request is a failed job item, not data corruption and not a user-facing primary path. A longer outage would degrade search to the lexical half and fail utility jobs — the evidence captured here does not show that.

Not established

  • Why the connection was refused. The ollama pod's state at that instant was not read; its {path} history for that window is the primary missing evidence.
  • Which portal consumer issued the request. The (unnamed) pipeline instance and the truncated stack do not name it — candidates are any of the OpenAICompatible chat consumers, the OllamaEmbeddingProvider, and the OllamaVision OCR engine.
  • Whether the caller recovered. Only the attempt-3 failure line was captured; a subsequent successful call or a permanent job failure is not in the evidence.
  • Whether this is the same condition as the morning's ReplicaShortfall or a different one (for example a normal restart gap: the chart deploys with strategy: Recreate, and the design page states callers absorb that gap as a failed call; the entrypoint also preloads ~23 GB of weights before the pod turns ready, so a restarting pod refuses connections for the whole load).

Candidate causes and the reading that separates them

  1. The ollama pod was restarting or not ready at that instant (Recreate roll, preload gate, readiness probe failing). Predicts: ollama pod {path} around the sighting show a container start, readiness failures, or OOM; portal lines stop being refused once the pod is ready. Reading: the ollama pod's restart and readiness-probe history for that window.
  2. The ReplicaShortfall condition recurring (0/1 ready again, e.g. unschedulable or OOMKilled pod). Predicts: the deployment's ready count is 0/1 {path} the sighting, the {path} reopens, and more occurrences appear in later windows. Reading: current replica state of the ollama deployment plus the following watcher window.
  3. {path} misconfiguration (release absent or namespace mismatch — no endpoints at all). Predicts a persistent condition with many occurrences across all portal replicas, not one line on one pod; the single sighting argues against it, but it is not excluded.

Where to look

  • {path} — the service contract, preload-then-ready behaviour, Recreate strategy.
  • {path} — the ChartRelease record for the ollama release on memex.
  • {path} — the earlier {path} outage on the same deployment, same morning.
  • The ollama pod's events and readiness-probe history for the sighting window, and the memex portal log around the same instant for the failing caller (the unnamed default client does not name it).
Evidence
Fingerprint 132bdddc7f9d98dd
Category Polly
Severity Error
Exception System.Net.Http.HttpRequestException
Namespace memex 1 occurrence(s), 2026-10-30 10:51:16Z → 2026-10-30 10:51:16Z; pods memex-portal-deployment-9655974b9-cqv54
Occurrences 1
First seen 2026-10-30 10:51:16Z
Last seen 2026-10-30 10:51:16Z
Routing not determined — no configured route matches the category Polly. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.
Recent log lines 
2026-10-30 10:51:16Z memex-portal-deployment-9655974b9-cqv54 fail: Polly[3]
      Execution attempt. Source: '-standard/(unnamed){path}', Operation Key: '', Result: 'Connection refused (ollama:11434)', Handled: 'True', Attempt: '3', Execution Time: 6.3231ms
      System.Net.Http.HttpRequestException: Connection refused (ollama:11434)
       ---> System.Net.Sockets.SocketException (111): Connection refused
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.InjectNewHttp11ConnectionAsync(QueueItem queueItem)
         at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionWaiter`1.WaitForConnectionWithTelemetryAsync(HttpRequestMessage request, HttpConnectionPool pool, Boolean async, CancellationToken requestCancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.Metri…[truncated]

This log site's previous ticket, https://github.com/Systemorph/MeshWeaver/issues/7553, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

Opened automatically from Admin/_LogIncident/132bdddc7f9d98dd. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 1108a4f93af857df: other fingerprints of this site fold in here as comments rather than opening tickets of their own.

Dominant language
C#
Stars
12
Forks
5
Avg merge
5h 10m
Merged PRs (30d)
981

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Systemorph/MeshWeaver

All issues in Systemorph/MeshWeaver

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.