ASGI app SIGTERM handler silently replaces the hosting server's handler
評估
研究方向
Read python/restate/server.py:254-270 and inspect how asgi_app registers the SIGTERM handler for request and lifespan scopes. Reproduce the displacement with the provided minimal repro, then check behavior with both loop.add_signal_handler and signal.signal. Done means the host server's shutdown handler still runs after restate.app(...) is called, while active receive channels are notified.
由索引模型根據 Issue 內容生成。
描述
On the first call into restate.app(...) of any scope, asgi_app registers loop.add_signal_handler(signal.SIGTERM, _on_sigterm) (python/restate/server.py:254-267). Because loop.add_signal_handler replaces the existing handler without chaining, SIGTERM no longer reaches the hosting ASGI server's shutdown path after the first request.
Observed behavior
Measured on restate-sdk 1.0.5 (main 63d2fe5), Python 3.13:
- Hypercorn 0.18: serving a plain ASGI app exits in ~0.1 s on SIGTERM. Serving
restate.app(...), hitting/healthonce, and sending SIGTERM leaves the process running past a 15 s grace period until SIGKILL. - Uvicorn 0.54.0: serving a plain ASGI app exits in ~0.1 s on SIGTERM. Serving
restate.app(...)after one request hangs the same way until SIGKILL. - Lifespan scope: a single
lifespan.startupcall is enough to displace the host handler, because the signal handler is installed before thescope["type"] == "lifespan"check (python/restate/server.py:269-270).
_on_sigterm only calls notify_shutdown() on active ReceiveChannel instances. Nothing notifies the host server to stop accepting connections or exit.
Why both loop.add_signal_handler and signal.signal matter
ASGI servers register SIGTERM in two ways on Unix:
- Uvicorn uses
signal.signal(signal.SIGTERM, ...). - Hypercorn uses
loop.add_signal_handler(signal.SIGTERM, ...)(hypercorn/asyncio/run.py). Onceloop.add_signal_handleris active, Python'ssignal.getsignal(signal.SIGTERM)returnsasyncio.unix_events._sighandler_noop, and the actual host callback lives inloop._signal_handlers[signal.SIGTERM](anasyncio.Handlestoring the callback and its arguments).
PR #226 adds re-dispatch using signal.getsignal(signal.SIGTERM) before loop.add_signal_handler. Tested on top of 1.0.5, #226 fixes Uvicorn, but Hypercorn still hangs for 15 s because signal.getsignal(signal.SIGTERM) only captures _sighandler_noop rather than the previous loop handler.
Suggested fix
Before calling loop.add_signal_handler(signal.SIGTERM, _on_sigterm):
- Check
getattr(loop, "_signal_handlers", {}).get(signal.SIGTERM)first. If present, save thatasyncio.Handleand invokeprev_handle._run()from_on_sigtermafter notifying the channels. - Otherwise, fall back to
signal.getsignal(signal.SIGTERM)and invoke it asprev(signal.SIGTERM, None)when callable and notasyncio.unix_events._sighandler_noop. - Optionally move the registration after the
scope["type"] == "lifespan"check so a rejected lifespan call does not touch process signals.
Minimal repro (no external server required)
# uv run --no-project --python 3.13 --with restate-sdk==1.0.5 python repro.py
import asyncio
import signal
import restate
calls: list[str] = []
def build_app():
svc = restate.Service("probe")
@svc.handler()
async def noop(ctx: restate.Context) -> str:
return "ok"
return restate.app(services=[svc])
async def one_call(app):
async def receive():
return {"type": "lifespan.startup"}
async def send(msg):
pass
try:
await app({"type": "lifespan", "asgi": {"version": "3.0"}}, receive, send)
except Exception:
pass
def run_case(style: str) -> bool:
calls.clear()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
if style == "loop.add_signal_handler":
loop.add_signal_handler(signal.SIGTERM, lambda: calls.append("host_ran"))
elif style == "signal.signal":
signal.signal(signal.SIGTERM, lambda *_: calls.append("host_ran"))
app = build_app()
loop.run_until_complete(one_call(app))
signal.raise_signal(signal.SIGTERM)
loop.run_until_complete(asyncio.sleep(0.05))
print(f"{style:<24} -> {calls or ['displaced (nothing ran)']}")
return "host_ran" in calls
for mode in ("loop.add_signal_handler", "signal.signal"):
run_case(mode)
Output on restate-sdk 1.0.5:
loop.add_signal_handler -> ['displaced (nothing ran)']
signal.signal -> ['displaced (nothing ran)']
- 主要語言
- Python
- 星號
- 83
- 分支
- 26
- 平均合併
- 7 天 4 小時
- 30 天內合併 PR
- 3
環境準備
這個專案沒有提供開發容器、Dockerfile 或貢獻指南,環境需要你自己搭建:先看它的 README,通用步驟見我們的新手貢獻指南。
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
restatedev/sdk-python 的其他 Issue
-
pydantic ext: request_stream() never initializes the turnstile — KeyError('call_...') on first tool call of a streamed run可能已有人在做 @QuocThuanTruong 於 91 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 76/100
restatedev/sdk-python#223 · 1 則留言 ·
-
難度 2/5 1-2 天 新手友好度 68/100
restatedev/sdk-python#204 · 1 則留言 ·
-
pydantic ext: tool results lose ToolReturn and deferred metadata, turnstile breaks on deferred and nested calls, iter() is not durable可能已有人在做 @sharosoo 於 8 天前認領。 未關閉
難度 4/5 3-5 天 新手友好度 48/100
restatedev/sdk-python#237 ·
-
RestateModelWrapper should not wrap `SdkInternalBaseException` as generic Exception; classify non-retryable provider errors as terminal可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉
難度 4/5 3-5 天 新手友好度 48/100
restatedev/sdk-python#225 ·
-
pydantic ext: auto_wrap_tools + async event_stream_handler crashes with 'Object of type coroutine is not JSON serializable' on first tool call可能已有人在做 @QuocThuanTruong 於 91 天前認領。 未關閉
難度 3/5 1-2 天 新手友好度 68/100
restatedev/sdk-python#222 · 1 則留言 ·
查看 restatedev/sdk-python 的全部 Issue
相似的 Issue
-
link-rot
難度 1/5 1 小時以內 新手友好度 67/100
-
難度 2/5 1-3 小時 新手友好度 82/100
developmentseed/titiler-cmr#265 ·
維護者通常 1 天內回覆
-
bug
難度 2/5 1-3 小時 新手友好度 72/100
維護者通常 1 天內回覆
-
Insecure Brotli version <1.2.1可能已有人在做 @jamescw19 今天認領。 未關閉
難度 1/5 1 小時以內 新手友好度 85/100
stac-utils/stac-fastapi-pgstac#438 · 1 個 reaction ·
-
bug triage
難度 2/5 1-3 小時 新手友好度 66/100
維護者通常 1 天內回覆