collections/batch: one object that fails client-side encoding fails every object sent with it
Maintainer thường phản hồi trong vòng 1 ngày
Đánh giá
- Độ khó
- 3/5
- Thời gian dự kiến
- 1-2 ngày
- Mức phù hợp với người mới
- 72/100
Hướng nghiên cứu
Start with weaviate/collections/batch/grpc_batch.py and its grpc_objects method, then read weaviate/collections/batch/base.py around __send_batch and weaviate/collections/batch/sync.py around __generate_stream_requests. Track how each encoding error is recorded in failed_objects and how remaining objects are sent. Done when one invalid object is reported individually and valid objects still reach the server in both batch modes.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
What happens
One object whose vector or property the client cannot encode fails every object batched with it, and the error does not say which object it was. In batch.stream() the objects are dropped without any exception.
fixed_size
Add 100 objects through collection.batch.fixed_size(batch_size=100), each with a colbert multi-vector. Object 37 carries the ragged [[0.1, 0.2], [0.3]].
- With #2039,
_Pack.multi, which packs a multi-vector into bytes, raisesWeaviateInvalidInputErroron object 37's vector. _BatchGRPC.grpc_objectsbuilds the protobuf message for all 100 objects in one list comprehension. The error aborts the request before anything is sent.- The
except Exceptionin_BatchBase.__send_batchattaches that one error to all 100 objects. - The log shows
Failed to send all objects in a batch of 100.collection.batch.failed_objectsholds 100 entries, each readingMulti-vector embeddings must have consistent dimensions. Expected dimension 2, got 1.The server receives 0 objects.
The message names neither object 37 nor colbert. Retrying failed_objects sends object 37 along with the other 99, so the retry fails the same way.
stream
__generate_stream_requests encodes each object with grpc_object inside the background send loop. The error ends that loop, so no object still unsent reaches the server. An add_object called after the loop has died raises the error. When the code finished adding before that, the with block exits normally. failed_objects is empty, and the only trace is one ERROR log line. In the reproduction below, 0 of 300 objects arrive.
The silent exit comes from a second defect: the stream context manager's __exit__ never re-raises the loop's exception. That needs its own fix, and this issue covers only the per-object encoding failure.
Reachability
Reachable from user input. Once #2039 merges, any batch.add_object call with a ragged multi-vector triggers it. On main today, a Decimal property value triggers the same failure in both modes, as #2263 reports for fixed_size.
Severity
High. In stream mode every unsent object is lost, with no exception and an empty failed_objects. In fixed-size mode nothing is lost silently, but 99 valid objects are reported failed under an error that is not theirs.
Where
weaviate/collections/batch/grpc_batch.py:94:grpc_objectsencodes every object in one comprehension. https://github.com/weaviate/weaviate-python-client/blob/4c919f668f639347cc48878d39735972018016ec/weaviate/collections/batch/grpc_batch.py#L94-L95weaviate/collections/batch/base.py:647:__send_batchgives the exception to every object of the batch. https://github.com/weaviate/weaviate-python-client/blob/4c919f668f639347cc48878d39735972018016ec/weaviate/collections/batch/base.py#L647-L656weaviate/collections/batch/sync.py:239: the stream loop encodes each object with no per-object catch. https://github.com/weaviate/weaviate-python-client/blob/4c919f668f639347cc48878d39735972018016ec/weaviate/collections/batch/sync.py#L238-L239weaviate/collections/grpc/shared.py:855: the ragged check #2039 adds. https://github.com/weaviate/weaviate-python-client/blob/96a0c28eb618cccdbfc5143c8a15adf17e1ce01b/weaviate/collections/grpc/shared.py#L853-L857
# grpc_batch.py:94
def grpc_objects(self, objects: List[_BatchObject]) -> List[batch_pb2.BatchObject]:
return [self.grpc_object(obj) for obj in objects]
# base.py:647
except Exception as e:
errors_obj = {
idx: ErrorObject(message=repr(e), object_=obj) for idx, obj in enumerate(objs)
}
logger.error(
{
"message": f"Failed to send all objects in a batch of {len(objs)}",
"error": repr(e),
}
)
# sync.py:238
for object_ in objects:
obj = self.__batch_grpc.grpc_object(object_._to_internal())
Suggested fix
Catch the error per object in grpc_objects and in __generate_stream_requests. Record it as that object's ErrorObject, the entry failed_objects returns, keyed by obj.index like the server-side errors. Then send the remaining objects. Put the object's UUID and the vector name in the message; __multi_vec already has the name when it calls _Pack.parse_single_or_multi_vec. After the fix, the reproduction below should print server received 99, failed_objects 1 for fixed_size and server received 299 for stream.
Reproduction
This runs offline against the mock servers in mock_tests/conftest.py. Save it as mock_tests/test_ragged_repro.py and run pytest mock_tests/test_ragged_repro.py -s.
import weaviate
from weaviate.proto.v1 import batch_pb2, weaviate_pb2_grpc
from .conftest import MOCK_IP, MOCK_PORT, MOCK_PORT_GRPC
from .mock_data import mock_class
def vector(i):
if i == 37:
return {"colbert": [[0.1, 0.2], [0.3]]} # ragged
return {"colbert": [[0.1, 0.2], [0.3, 0.4]]}
class Service(weaviate_pb2_grpc.WeaviateServicer):
received = 0
def BatchObjects(self, request, context):
Service.received += len(request.objects)
return batch_pb2.BatchObjectsReply()
def BatchStream(self, request_iterator, context):
yield batch_pb2.BatchStreamReply(started=batch_pb2.BatchStreamReply.Started())
for request in request_iterator:
if request.HasField("data"):
uuids = [o.uuid for o in request.data.objects.values]
Service.received += len(uuids)
yield batch_pb2.BatchStreamReply(acks=batch_pb2.BatchStreamReply.Acks(uuids=uuids))
if request.HasField("stop"):
return
def connect(weaviate_mock, grpc_server):
Service.received = 0
weaviate_pb2_grpc.add_WeaviateServicer_to_server(Service(), grpc_server)
weaviate_mock.expect_request(f"/v1/schema/{mock_class['class']}").respond_with_json(mock_class)
client = weaviate.connect_to_local(port=MOCK_PORT, host=MOCK_IP, grpc_port=MOCK_PORT_GRPC)
return client, client.collections.use(mock_class["class"])
def test_fixed_size(weaviate_mock, start_grpc_server):
client, col = connect(weaviate_mock, start_grpc_server)
with col.batch.fixed_size(batch_size=100) as batch:
for i in range(100):
batch.add_object({"name": f"Object {i}"}, vector=vector(i))
failed = col.batch.failed_objects
print(f"\nfixed_size: server received {Service.received}, failed_objects {len(failed)}")
print(f"fixed_size: messages {({f.message for f in failed})}")
client.close()
def test_stream(weaviate_mock, start_grpc_server):
client, col = connect(weaviate_mock, start_grpc_server)
with col.batch.stream() as batch:
for i in range(300):
batch.add_object({"name": f"Object {i}"}, vector=vector(i))
failed = col.batch.failed_objects
print(f"\nstream: no exception, server received {Service.received}, failed_objects {len(failed)}")
client.close()
On #2039's head, 96a0c28e:
fixed_size: server received 0, failed_objects 100
fixed_size: messages {"WeaviateInvalidInputError('Invalid input provided: Multi-vector embeddings must have consistent dimensions. Expected dimension 2, got 1..')"}
stream: no exception, server received 0, failed_objects 0
On main, 4c919f66:
fixed_size: server received 100, failed_objects 0
fixed_size: messages set()
stream: no exception, server received 300, failed_objects 0
Provenance
Aggravated by #2039. The whole-batch failure predates it. On main at 4c919f66, one Decimal value already fails the batch in fixed_size (#2263) and drops 300 of 300 objects in stream. #2039 adds ragged multi-vectors as a new way to reach it, and the fix lands outside its diff. Determined against 4c919f66...96a0c28e.
Verification
Confirmed. The reproduction above ran on #2039's head and on main, with the outputs shown. The Decimal stream case ran on main against the same mock service.
Where it surfaced
Review of #2039 (pragnyan/reject-ragged-multi-vectors) against main at 4c919f66. Related: #2263 reports the same failure through a Decimal property; its author plans a fix for the error message and left the whole-batch failure open. #2165 reports how __send_batch keys these whole-batch errors.
- Ngôn ngữ chính
- Python
- Star
- 230
- Fork
- 173
- Merge trung bình
- 4 ngày 16 giờ
- Pull request đã merge (30 ngày)
- 23
Chuẩn bị môi trường
- Không có Dockerfile hay tệp Docker Compose
- Không có mẫu pull request
- Đọc hướng dẫn đóng góp
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của weaviate/weaviate-python-client
-
bug
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 84/100
weaviate/weaviate-python-client#2265 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Batch: re-added rate-limited objects stay in results.objs.all_responses as errorsCó thể đã có người làm Có pull request liên kết đang mở hoặc đã được merge. Đang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
weaviate/weaviate-python-client#2179 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
batch insert fails the whole batch with "ValueError: Unexpected type" when a value is a DecimalCó thể đã có người làm @steven-asearcy đã nhận hôm nay. Đang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 35/100
weaviate/weaviate-python-client#2263 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
collections/batch/sync.py: after a fast hang-up reconnect the old stream's sender steals a request, hanging add_objectCó thể đã có người làm @shadowmodder đã nhận 2 ngày trước. Đang mởbug
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 45/100
weaviate/weaviate-python-client#2261 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 48/100
weaviate/weaviate-python-client#2259 ·
Maintainer thường phản hồi trong vòng 1 ngày
Tất cả issue của weaviate/weaviate-python-client
Issue tương tự
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 67/100
-
breaking change
Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 84/100
it-at-m/lhm_actions#441 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 62/100
TigreGotico/euskaphone#68 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
[BUG] arc_agi_2: a correct grid in the format the prompt asks for scores 0 (exact_match gold is a one-line list)Có thể đã có người làm @BahaaMattar đã nhận hôm nay. Đang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
huggingface/lighteval#1510 ·
Maintainer thường phản hồi trong vòng 1 ngày