collections/batch/sync.py: after a fast hang-up reconnect the old stream's sender steals a request, hanging add_object

未关闭
#2,261 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@shadowmodder 已经在做这个了。

开始于 2026年11月16日。

  • #2262 来自 @shadowmodder —— 未关闭

评估

难度
4/5
预计耗时
3-5 天
新手友好度
45/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
grpc, python

调研方向

Start with weaviate/collections/batch/sync.py, especially __send around line 294 and the reconnect logic in recv_wrapper around lines 520–532. Trace how a request taken from __reqs is handled when its stream has ended, then add a regression test using a stubbed server that reconnects quickly. Done means queued objects are neither lost nor left inflight, and shutdown still ends both stream threads.

由索引模型根据 Issue 内容生成。

描述

bug

What goes wrong

During collection.batch.stream(), the server can end the gRPC stream unexpectedly ("context canceled" or "Socket closed", as when the coordinator node shuts down ungracefully). The client then reconnects and opens a new stream. If that reconnect finishes in under a second, the old stream's request sender is still running. It takes the next batch off the queue and hands it to the dead stream, where gRPC drops it. Those objects never reach the server and never show up in failed_objects, and every later add_object call hangs.

How the pieces work today:

  • __send (sync.py:274) is the generator gRPC reads requests from. It waits up to 1 s for a request (self.__reqs.get(timeout=1), sync.py:294). It checks the hung-up, shutting-down and out-of-memory flags, and stops, only when that wait times out (sync.py:306-315).
  • On a hang-up, recv_wrapper reconnects (sync.py:524), clears the sets of unacknowledged uuids (sync.py:529-530), and starts __recv again. The new __recv first clears the hung-up flag (sync.py:322-324).
  • gRPC's thread that reads requests for the old call stays blocked inside __send. When it gets a request, it sees the call has ended and returns without sending it (consume_request_iterator in grpc/_channel.py).
  • add_object waits while __is_blocked (sync.py:636-643) is true. That includes when __inflight_objs, the set of sent but unacknowledged object uuids, holds __batch_size entries or more.

One run, with a batch size of 100:

  1. A stream import is running. The old __send is waiting in get(timeout=1).
  2. The server hangs up. recv_wrapper sets the hung-up flag, reconnects in 300 ms, clears __inflight_objs, and starts a new __recv, which clears the hung-up flag.
  3. The old __send's wait has not timed out, so it never saw the flag. __loop puts the next 100 objects on the queue, and the old __send takes them. It adds their 100 uuids to __inflight_objs (sync.py:301-303) and yields the request into the dead call, which drops it.
  4. The server never receives those 100 objects, so it never acknowledges them. __inflight_objs stays at 100, __is_blocked stays true, and the next add_object waits forever. failed_objects is empty.

When the old sender takes the None that _shutdown puts on the queue instead, the new stream never gets its stop message and its receive thread never ends. _wait's join then times out without raising.

Reachability: reachable. The trigger is a server hang-up on a multi-node cluster at a consistency level other than ALL. There, __reconnect (sync.py:468-492) calls close() and connect(force=True), with no node-health wait and no sleep when the first attempt succeeds. A single-node cluster or ConsistencyLevel.ALL first waits for every node to report healthy, polling every 5 s. A failed first attempt sleeps 1 s. Either delay lets the old get time out and see the flag, which closes the window. Not measured: how long a production close() plus connect(force=True) takes, which decides how often the window opens.

Severity: high. Up to one batch of objects is lost without appearing in failed_objects, and the import hangs.

Location: weaviate/collections/batch/sync.py:294 at origin/main 2ec127a3. The other lines cited above are at the same commit.

Excerpt (sync.py:293-324 at 2ec127a3, middle cut):

            try:
                req = self.__reqs.get(timeout=1)
                if req is None:
                    ...
                    yield batch_pb2.BatchStreamRequest(stop=batch_pb2.BatchStreamRequest.Stop())
                    return
                with self.__acks_lock:
                    self.__inflight_objs.update(req.uuids)
                    self.__inflight_refs.update(req.beacons)
                yield req.proto
                continue
            except Empty:
                if self.__is_shutting_down.is_set():
                    ...
                    return
                elif self.__is_oom.is_set():
                    ...
                    return
                elif self.__is_hungup.is_set():
                    logger.info("Detected hung up stream, closing the client-side of the stream")
                    return
    ...

    def __recv(self) -> None:
        self.__is_renewing_stream.clear()
        self.__is_shutting_down.clear()
        self.__is_hungup.clear()

Reproduction: no committed test. A probe against a stubbed server reproduced it on two runs, with the reconnect duration stubbed as well:

  • Stream 1 aborted at start, instant reconnect: added=100 server got=0 lost=100 failed_objects=0 inflight=100. The user thread was still stuck in _add_object after 25 s.
  • Control, 2 s reconnect: server got=500 lost=0, and both background threads end.

Provenance: pre-existing. Blame dates the cited lines to commits from January to April 2026 (ab084109, 66407ec5, 06726fe9, fbbaae8d, 6dce1053). They are still on origin/main at 2ec127a3.

Suggested fix: give each stream its own stop signal, or a generation counter, that the old __send checks after get. It then puts a taken request back, or fails it, instead of yielding it into a dead call.

Related: #2139 is about the same reconnect path but a different defect: there, __reconnect closes the connection that every thread shares.

主要语言
Python
星标
230
派生
173
平均合并
4 天 16 小时
30 天内合并 PR
23

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

weaviate/weaviate-python-client 的其他 Issue

查看 weaviate/weaviate-python-client 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。