Investigate intermittent staging Copilot messages first-event timeouts

Open
  • Is something wrong?
#3,363 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@nighca is already working on this.

Since Jul 22, 2026.

Assessment

This issue has not been assessed yet.

Description

Background

The staging Copilot POST /copilot/messages endpoint occasionally approaches or exceeds the first-event timeout. The Sentry chart for Copilot messages first event wait P95 shows intermittent staging spikes, including one close to 20 seconds.

An important comparison is that production remains normal during the same staging slow periods, although both environments use the same QNAIGC service endpoint. This makes a global QNAIGC incident alone insufficient to explain the symptom and requires environment-aware evidence.

Evidence screenshot

https://goplus-xk.sentry.io/dashboard/8546578/?project=4509761000308736&project=4509868947800064&statsPeriod=7d

Image

Scope

Locate the source of intermittent first-event latency in staging and determine whether it is caused by:

  • an intermittent QNAIGC provider-side failure or slowdown;
  • an issue in the network environment hosting builder-backend; or
  • a delay inside the backend or its proxy/streaming path after the upstream response begins.

Investigation plan

  • Correlate slow staging requests with equivalent production requests over the same time windows, model/provider configuration, and QNAIGC endpoint.
  • Add structured telemetry around the outbound QNAIGC request and SSE relay, sufficient to separate DNS/connect/TLS/request-write/response-header/first-upstream-event/first-client-flush timing and classify errors or cancellations.
  • Record non-sensitive environment and request correlation fields needed to compare staging and production without logging prompts, response content, credentials, or personal data.
  • Correlate the new signals with ingress/proxy and host/network telemetry where available.
  • Use traces from slow and normal requests to decide whether the dominant wait is provider-side, network-path-specific, or backend/proxy-side.

Expected outcome

  • We can attribute each slow segment with enough confidence to choose the next owner and remediation.
  • The added reporting makes future staging-only spikes diagnosable against the production control group.
  • Any confirmed configuration, network, provider, or code issue has a separately actionable follow-up or fix.

Related

  • Closed predecessor: #3334
Dominant language
TypeScript
Stars
70
Forks
59
Avg merge
1d 17h
Merged PRs (30d)
69

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from goplus/builder

All issues in goplus/builder

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.