ASGI app SIGTERM handler silently replaces the hosting server's handler
Bewertung
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Anfängerfreundlichkeit
- 52/100
Rechercherichtung
Read python/restate/server.py:254-270 and inspect how asgi_app registers the SIGTERM handler for request and lifespan scopes. Reproduce the displacement with the provided minimal repro, then check behavior with both loop.add_signal_handler and signal.signal. Done means the host server's shutdown handler still runs after restate.app(...) is called, while active receive channels are notified.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
On the first call into restate.app(...) of any scope, asgi_app registers loop.add_signal_handler(signal.SIGTERM, _on_sigterm) (python/restate/server.py:254-267). Because loop.add_signal_handler replaces the existing handler without chaining, SIGTERM no longer reaches the hosting ASGI server's shutdown path after the first request.
Observed behavior
Measured on restate-sdk 1.0.5 (main 63d2fe5), Python 3.13:
- Hypercorn 0.18: serving a plain ASGI app exits in ~0.1 s on SIGTERM. Serving
restate.app(...), hitting/healthonce, and sending SIGTERM leaves the process running past a 15 s grace period until SIGKILL. - Uvicorn 0.54.0: serving a plain ASGI app exits in ~0.1 s on SIGTERM. Serving
restate.app(...)after one request hangs the same way until SIGKILL. - Lifespan scope: a single
lifespan.startupcall is enough to displace the host handler, because the signal handler is installed before thescope["type"] == "lifespan"check (python/restate/server.py:269-270).
_on_sigterm only calls notify_shutdown() on active ReceiveChannel instances. Nothing notifies the host server to stop accepting connections or exit.
Why both loop.add_signal_handler and signal.signal matter
ASGI servers register SIGTERM in two ways on Unix:
- Uvicorn uses
signal.signal(signal.SIGTERM, ...). - Hypercorn uses
loop.add_signal_handler(signal.SIGTERM, ...)(hypercorn/asyncio/run.py). Onceloop.add_signal_handleris active, Python'ssignal.getsignal(signal.SIGTERM)returnsasyncio.unix_events._sighandler_noop, and the actual host callback lives inloop._signal_handlers[signal.SIGTERM](anasyncio.Handlestoring the callback and its arguments).
PR #226 adds re-dispatch using signal.getsignal(signal.SIGTERM) before loop.add_signal_handler. Tested on top of 1.0.5, #226 fixes Uvicorn, but Hypercorn still hangs for 15 s because signal.getsignal(signal.SIGTERM) only captures _sighandler_noop rather than the previous loop handler.
Suggested fix
Before calling loop.add_signal_handler(signal.SIGTERM, _on_sigterm):
- Check
getattr(loop, "_signal_handlers", {}).get(signal.SIGTERM)first. If present, save thatasyncio.Handleand invokeprev_handle._run()from_on_sigtermafter notifying the channels. - Otherwise, fall back to
signal.getsignal(signal.SIGTERM)and invoke it asprev(signal.SIGTERM, None)when callable and notasyncio.unix_events._sighandler_noop. - Optionally move the registration after the
scope["type"] == "lifespan"check so a rejected lifespan call does not touch process signals.
Minimal repro (no external server required)
# uv run --no-project --python 3.13 --with restate-sdk==1.0.5 python repro.py
import asyncio
import signal
import restate
calls: list[str] = []
def build_app():
svc = restate.Service("probe")
@svc.handler()
async def noop(ctx: restate.Context) -> str:
return "ok"
return restate.app(services=[svc])
async def one_call(app):
async def receive():
return {"type": "lifespan.startup"}
async def send(msg):
pass
try:
await app({"type": "lifespan", "asgi": {"version": "3.0"}}, receive, send)
except Exception:
pass
def run_case(style: str) -> bool:
calls.clear()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
if style == "loop.add_signal_handler":
loop.add_signal_handler(signal.SIGTERM, lambda: calls.append("host_ran"))
elif style == "signal.signal":
signal.signal(signal.SIGTERM, lambda *_: calls.append("host_ran"))
app = build_app()
loop.run_until_complete(one_call(app))
signal.raise_signal(signal.SIGTERM)
loop.run_until_complete(asyncio.sleep(0.05))
print(f"{style:<24} -> {calls or ['displaced (nothing ran)']}")
return "host_ran" in calls
for mode in ("loop.add_signal_handler", "signal.signal"):
run_case(mode)
Output on restate-sdk 1.0.5:
loop.add_signal_handler -> ['displaced (nothing ran)']
signal.signal -> ['displaced (nothing ran)']
- Vorherrschende Sprache
- Python
- Sterne
- 83
- Forks
- 26
- Ø Merge
- 7 T. 4 Std.
- Gemergte PRs (30 T.)
- 3
Entwicklungsumgebung
Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus restatedev/sdk-python
-
pydantic ext: request_stream() never initializes the turnstile — KeyError('call_...') on first tool call of a streamed runEvtl. vergeben @QuocThuanTruong hat das vor 88 Tagen übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
restatedev/sdk-python#223 · 1 Kommentar ·
-
Schwierigkeit 2/5 1-2 Tage Anfängerfreundlichkeit 68/100
restatedev/sdk-python#204 · 1 Kommentar ·
-
pydantic ext: tool results lose ToolReturn and deferred metadata, turnstile breaks on deferred and nested calls, iter() is not durableEvtl. vergeben @sharosoo hat das vor 5 Tagen übernommen. Offen
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 48/100
restatedev/sdk-python#237 ·
-
RestateModelWrapper should not wrap `SdkInternalBaseException` as generic Exception; classify non-retryable provider errors as terminalEvtl. vergeben Ein verknüpfter Pull Request ist offen oder bereits gemergt. Offen
Schwierigkeit 4/5 3-5 Tage Anfängerfreundlichkeit 48/100
restatedev/sdk-python#225 ·
-
pydantic ext: auto_wrap_tools + async event_stream_handler crashes with 'Object of type coroutine is not JSON serializable' on first tool callEvtl. vergeben @QuocThuanTruong hat das vor 88 Tagen übernommen. Offen
Schwierigkeit 3/5 1-2 Tage Anfängerfreundlichkeit 68/100
restatedev/sdk-python#222 · 1 Kommentar ·
Alle Issues in restatedev/sdk-python
Ähnliche Issues
-
style: article/section 📝 system: cloud.rc ☁️ system: security 🔐
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 65/100
uabrc/uabrc.github.io#1262 ·
-
documentation
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 70/100
NCAR/miles-credit#526 ·
-
documentation
Schwierigkeit 1/5 1-3 Stunden Anfängerfreundlichkeit 76/100
conda/conda-lock#957 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Type: Bug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 65/100
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
aws-samples/sample-per-model-bedrock#464 ·
Maintainer antworten meist innerhalb von 1 Tag