Batch: re-added rate-limited objects stay in results.objs.all_responses as errors
维护者通常 1 天内回复
评估
调研方向
从 weaviate/collections/batch/base.py 中的 _BatchBase.__send_batch 开始,并使用 mock gRPC fixtures 通过 mock_tests/test_probe_readd_dep020.py 重现。验证 retry 会移除过时的 ErrorObject,不会发出 Dep020 警告,并在不读取已弃用的 all_responses 的情况下留下四个结果条目。
由索引模型根据 Issue 内容生成。
描述
A rate-limited batch emits Dep020 in code that never reads all_responses, because the re-add in _BatchBase.__send_batch reads the deprecated property (weaviate/collections/batch/base.py:720). That line also filters by list position, while errors and uuids filter by the batch-wide index of the object. From the second chunk on the two differ, and the ErrorObject of the re-added object stays in _all_responses after its retry succeeds.
Mock gRPC server at 38f30c1 with the mock_tests/conftest.py fixtures, 4 objects through fixed_size(batch_size=2, concurrent_requests=1), the first object of the second chunk rate-limited once. The test never reads all_responses:
$ pytest mock_tests/test_probe_readd_dep020.py -q -s
Dep020 during batch=1 errors=0 len(_all_responses)=5 stale=1
mock_tests/test_probe_readd_dep020.py
import warnings
import weaviate
from weaviate.collections.classes.batch import ErrorObject
from weaviate.proto.v1 import batch_pb2, weaviate_pb2_grpc
from .conftest import MOCK_IP, MOCK_PORT, MOCK_PORT_GRPC, mock_class
class Svc(weaviate_pb2_grpc.WeaviateServicer):
calls = 0
def BatchObjects(self, request, context):
self.calls += 1
if self.calls == 2: # second chunk, once: rate-limit its first object
return batch_pb2.BatchObjectsReply(errors=[{"index": 0, "error": "failed with status: 503 error"}])
return batch_pb2.BatchObjectsReply()
def test_readd(weaviate_mock, start_grpc_server):
weaviate_mock.expect_request(f"/v1/schema/{mock_class['class']}").respond_with_json(mock_class)
weaviate_pb2_grpc.add_WeaviateServicer_to_server(Svc(), start_grpc_server)
client = weaviate.connect_to_local(port=MOCK_PORT, host=MOCK_IP, grpc_port=MOCK_PORT_GRPC)
try:
col = client.collections.use(mock_class["class"])
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter("always")
with col.batch.fixed_size(batch_size=2, concurrent_requests=1) as b:
for i in range(4):
b.add_object({"name": f"o{i}"})
r = col.batch.results.objs
dep = [x for x in w if "all_responses" in str(x.message)]
stale = [x for x in r._all_responses if isinstance(x, ErrorObject)]
print(f"\nDep020 during batch={len(dep)} errors={len(r.errors)} "
f"len(_all_responses)={len(r._all_responses)} stale={len(stale)}")
finally:
client.close()
Expected: no warning and 4 entries. Reading _all_responses and filtering on err.object_.index fixes both. #2166 already edits this function.
@tsmith023, you added Dep020. Would you rather take that one-line fix, or remove all_responses now that its Q4 2024 date has passed?
- 主要语言
- Python
- 星标
- 228
- 派生
- 167
- 平均合并
- 2 天 9 小时
- 30 天内合并 PR
- 14
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
weaviate/weaviate-python-client 的其他 Issue
-
insert_many silently discards vectors if they're NumPy arrays可能已有人在做 @tibor-reiss 于 828 天前认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 68/100
weaviate/weaviate-python-client#1002 · 5 条评论 ·
维护者通常 1 天内回复
-
error when client connect可能已有人在做 @dudanogueira 于 3 天前认领。 未关闭
难度 4/5 1-2 天 新手友好度 45/100
weaviate/weaviate-python-client#2220 · 2 条评论 ·
维护者通常 1 天内回复
-
Running backups and exports report completed_at as year 1 instead of None可能已有人在做 @RaphaelFakhri 于 4 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 30/100
weaviate/weaviate-python-client#2216 ·
维护者通常 1 天内回复
-
Client-side batching: wholesale chunk failures are reported under chunk-local indices, so errors collide and describe the wrong objects可能已有人在做 @feiiiiii5 于 36 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 15/100
weaviate/weaviate-python-client#2165 ·
维护者通常 1 天内回复
-
[HTTPXodus] Consider migrating from httpx to httpx2 (the actively maintained fork)可能已有人在做 @SiAgentBuild 于 51 天前认领。 未关闭
难度 5/5 一周以上 新手友好度 30/100
weaviate/weaviate-python-client#2154 ·
维护者通常 1 天内回复
查看 weaviate/weaviate-python-client 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 78/100
conda-forge/conda-forge-bot#7025 ·
维护者通常 1 天内回复
-
area:ci
难度 1/5 1 小时以内 新手友好度 74/100
catincloud-labs/dagster-dex#149 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 84/100
home-assistant/core#188424 ·
维护者通常 1 天内回复
-
Review backlog (docs-currency): README 'CI run' link lands on a run whose overall badge is cancelled未关闭agent/reviewer needs-kind needs-triage
难度 2/5 1-3 小时 新手友好度 63/100
projectbluefin/utah#880 · 2 条评论 ·
维护者通常 1 天内回复
-
LSan reports obj_gen leak on post-parse usage() exits (aarch64)可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 2/5 1-3 小时 新手友好度 76/100
redis/memtier_benchmark#850 ·
维护者通常 1 天内回复