WebMVC: upstream aborting the response body is logged by Tomcat as an unhandled ERROR and dispatched to `/error`

Aperta
  • Qualcosa non va?
#4,357 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, spring-boot
Ambito
backend

Direzione di ricerca

Start in AbstractProxyExchange.copyResponseBodyWithFlushing and copyResponseBody, where the upstream read failure is rethrown, and RestClientProxyExchange.doExchange, where the copy is invoked. Distinguish an upstream read failure from a client write failure, then decide how to abort the downstream response without propagating. The reproducer is a raw-socket test that sends a 200 text/event-stream, one event, then a TCP RST; done when the truncated stream still reaches the client and neither the Tomcat ERROR nor the /error dispatch is logged.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

waiting-for-triage

Describe the bug

Spring Cloud 2025.1.4 / spring-cloud-gateway-server-webmvc 5.0.4 (Spring Boot 4.0.9, Tomcat 11.0.27). The same behaviour occurs on 5.1.0-RC1 and on 4.1.x.

We run a WebMVC gateway in front of SSE (LLM streaming) backends. When a backend hits its own read timeout, it gives up on a stream it has already started: the status, headers and some events are already sent, and then it closes the connection without the terminating chunk. That's an everyday condition for us and nothing the gateway can fix. Even so, every occurrence shows up as an unhandled servlet exception:

ERROR o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception
java.io.IOException: closed
	at o.s.c.g.server.mvc.common.AbstractProxyExchange.copyResponseBodyWithFlushing(AbstractProxyExchange.java:97)
	at o.s.c.g.server.mvc.common.AbstractProxyExchange.copyResponseBody(AbstractProxyExchange.java:50)
	at o.s.c.g.server.mvc.handler.RestClientProxyExchange.lambda$doExchange$3(RestClientProxyExchange.java:82)
	at o.s.c.g.server.mvc.handler.GatewayServerResponseBuilder$WriteFunctionResponse.writeToInternal(GatewayServerResponseBuilder.java:232)
	at o.s.c.g.server.mvc.handler.AbstractGatewayServerResponse.writeTo(AbstractGatewayServerResponse.java:97)
	at o.s.web.servlet.function.support.HandlerFunctionAdapter.handle(HandlerFunctionAdapter.java:110)
	at o.s.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
	...
Caused by: java.io.IOException: chunked transfer encoding, state: READING_LENGTH
Caused by: java.net.SocketException: Connection reset

DEBUG o.s.web.servlet.DispatcherServlet : "INCLUDE" dispatch for POST "/error", parameters={}
WARN  .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.http.converter.HttpMessageNotWritableException: No converter for [class java.util.LinkedHashMap] with preset Content-Type 'text/event-stream']

What the client sees is already correct: a truncated chunked stream with no terminating chunk, so it can tell the stream didn't complete. The problem is the server side. An expected upstream condition produces one ERROR with a full stack trace and two more lines from an /error dispatch that can't write anything to a committed response. In our case this is the main source of noise in our error alerting.

As far as we can tell, there's no clean place to handle this in an application:

  • The upstream body is read inside ServerResponse.writeTo, after every HandlerFilterFunction has returned, so a route filter's try/catch around next.handle(request) never sees the exception.
  • Replacing MvcUtils.CLIENT_RESPONSE_INPUT_STREAM_ATTR from an outer filter doesn't work either, because the request is rebuilt with a copied attributes map.
  • Spring Framework's DisconnectedClientHelper handles the opposite case, where the downstream client went away. It matches on messages such as "connection reset", so it would also misclassify this upstream reset as a client disconnect.

Sample

A self-contained reproducer (single test, mvn test, JDK 21+): https://gist.github.com/Shawyeok/8bb7604a03b63598d0946a6a05dcd9f2

The test's upstream is a raw socket that sends 200 text/event-stream, one event, and then a TCP RST. The test asserts that the downstream stream is truncated, which passes, and that neither the Tomcat ERROR nor the /error dispatch appears, which fails.

What we do today (workaround)

We wrap the route's ServerResponse in a delegating GatewayServerResponse. Its writeTo hands SCG an HttpServletResponseWrapper that records whether a write or flush to the client failed:

  • If the client write failed, we rethrow unchanged, as today.
  • Otherwise the upstream read failed. We log one INFO line, increment a metric, and rethrow as org.apache.coyote.CloseNowException. Tomcat logs that only at DEBUG and still closes the connection without the terminating chunk.

We also exclude ErrorMvcAutoConfiguration, since a gateway has no error page of its own to serve.

This works, but it depends on a Tomcat-internal exception type and on wrapping every route, so we'd rather not keep it.

Questions for the maintainers

  1. Is it intended that an upstream I/O failure during the body copy propagates to the servlet container as an unhandled exception? Or would you consider handling it inside the gateway, for example by telling an upstream read failure apart from a client write failure in copyResponseBody, logging the former at a lower level, and aborting the downstream response without rethrowing?
  2. If you'd rather keep it propagating, could it become an exception type or hook that applications can recognise and handle? A dedicated exception, a callback, or a property would all work for us. Is there an approach you'd recommend instead of ours?
  3. For the /error dispatch on an already-committed proxied response: is that something the gateway should avoid, or is it Spring Boot / Tomcat behaviour that we should raise there?

Related: #3999 / #4217 changed how the upstream response is closed when the copy fails because the client disconnected. The upstream-side failure described here still rethrows on 5.0.4 and 5.1.0-RC1.

Happy to contribute a PR if you can point us at the direction you'd prefer.

Lingua principale
Java
Stelle
4.9k
Fork
3.5k
Merge medio
2g 4h
PR unite (30g)
9

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di spring-cloud/spring-cloud-gateway

Tutte le issue di spring-cloud/spring-cloud-gateway

Issue simili

Altre issue su Java

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.