qdrant/qdrant

Verify snapshot integrity with checksum in shard snapshot transfer

开放

#3,372 创建于 2024年1月10日

 (6 条评论) (1 个反应) (0 位负责人)Rust (2,263 个派生)batch import
enhancementgood first issue

仓库指标

星标
 (31,321 个星标)
PR 合并指标
 (平均合并 2天 20小时) (30 天内合并 276 个 PR)

描述

Depends on https://github.com/qdrant/qdrant/issues/3371.

Is your feature request related to a problem? Please describe. In https://github.com/qdrant/qdrant/pull/2840 we've added checksums for snapshot files. The implementation is somewhat limited however, and requires further integration to make full use of it.

Since Qdrant 1.7 different shard transfer methods are supported. Snapshot transfers have been added to make these transfers more capable by utilizing snapshots.

One problem with this approach is that we have no integrity checks for the actual snapshot files. If such file would become corrupted, Qdrant will happily restore possibly resulting in a broken shard.

Describe the solution you'd like When a shard snapshot transfer happens, we should check integrity of the snapshot file by verifying the attached checksum. Since https://github.com/qdrant/qdrant/pull/2840, the checksum is attached to the SnapshotDescription object.

https://github.com/qdrant/qdrant/issues/3371 will implement a checksum field in snapshot recovery endpoints. We'll have to wait for this to be implemented so that we can utilize this in the snapshot transfer progress.

The right approach is probably to pass the checksum along in the recovery call here: https://github.com/qdrant/qdrant/blob/c6a351c82d57eb786b75dac785de7b085c6f80ff/lib/collection/src/shards/transfer/snapshot.rs#L215-L228

If checksum verification on the remote node fails, we should clean up the snapshot file and return with an error. Cleaning up on the remote is probably already handled with https://github.com/qdrant/qdrant/issues/3371.

Additional context There's other work to be done to properly integrate checksums, but that will be handled in different issues/PRs.

贡献者指南