本地向量引擎 scroll 忽略 offset 参数，导致记录数>1 的 URI 覆盖写永久 500

Open
  • Is something wrong?
#7,468 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@lester076 is already working on this.

Since Nov 11, 2026.

  • #7471 by @lester076 — open

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
python
Domain
backend, database

Research direction

Start with openviking/storage/viking_vector_index_backend.py: read scroll() around lines 1055–1077 and _strict_scan() around lines 1822–1874, then trace how the local vector engine handles strict_query ordering and offset. Reproduce the repeated-page behavior with the debug scroll endpoint described in the issue. Done means pagination reaches distinct records and URI replacement succeeds; the linked pull request #7471 is already open.

Written by the indexing model from the issue text.

Description

环境

  • openviking v0.5.4.1（v0.5.4 同样复现）
  • 自建单节点，Docker，Linux x86_64，local vectordb backend
  • 单 collection context，index default，1024 维
  • 数据规模：53,003 条向量记录

现象

对记录数 > 1 的 URI 执行覆盖写（POST /api/v1/content/write，mode=replace）持续返回 HTTP 500，且不可自愈：

Incremental inventory under viking://.../README.md was incomplete; retrying full inventory attempt=1/3
  error=Incremental inventory under viking://.../README.md ended after 0 of 1 records
...
openviking.server.app - ERROR - [request_id=...] Unhandled exception
  File ".../openviking/server/routers/content.py", line 296, in write
RuntimeError: Incremental inventory under viking://.../README.md ended after 0 of 1 records

关键特征：删除该 URI 后首次写入成功，第二次（覆盖写）必失败，形成"假恢复"。

根因

openviking/storage/viking_vector_index_backend.py:1055-1077 的 scroll() 用 offset 游标分页：

async def scroll(self, filter=None, limit=100, cursor=None, output_fields=None):
    offset = int(cursor) if cursor else 0
    records = await self.strict_query(
        filter=filter, limit=limit, offset=offset, output_fields=output_fields,
        order_by="updated_at", order_desc=False,
    )
    next_cursor = str(offset + len(records)) if len(records) == limit else None
    return records, next_cursor

_strict_scan()（同文件 1822-1874）先取 strict_count（bitmap 聚合）作为 expected_count，再逐页 scroll；页空或游标耗尽即抛 ended after X of Y records。

实测证明：本地向量引擎忽略了 offset 参数。

可复现证据（只读 debug 接口）

# A) offset 是否生效（limit=5，offset=0 vs offset=5 应返回不同记录）
GET /api/v1/debug/vector/scroll?limit=5&offset=0
  -> ['d35311db','f41eb525','e04cdfa1','a8c1ede4','dcbc1895']
GET /api/v1/debug/vector/scroll?limit=5&offset=5
  -> ['d35311db','f41eb525','e04cdfa1','a8c1ede4','dcbc1895']
identical: True          # offset 被完全忽略

# B) 对照组：limit 生效，排除接口整体失效
limit=1   -> returned 1 records
limit=5   -> returned 5 records
limit=50  -> returned 50 records

# C) 分页穷尽测试（limit=200，翻 400 页）
collection total count: 53003
scroll reached: 80000  unique ids: 200  pages: 400
gap: 52803               # 永远只能拿到第一页的 200 条

失败链路

  1. strict_count 走 bitmap 聚合，正确返回 expected=2（旧 + 新记录）
  2. scroll 第 1 页（offset=0）返回 1 条，next_cursor="1"
  3. 第 2 页（offset=1）又返回同一条，去重后 scanned_count 仍为 1
  4. 游标耗尽时 scanned_count(1) != expected_count(2) → 抛 cursor ended after 1 of 2 records
  5. get_incremental_inventory_under_uri 重试 3 次全失败 → 向上抛 → content.write 500

为什么"删除后首次写回"能成功

删除后该 URI 只剩 1 条记录，expected=1，第一页即拿满，无需翻页 → 通过。
一旦覆盖写产生第 2 条（旧 + 新），必须翻页 → 必然失败。

影响面

  • 所有记录数 > 1 的 URI 的覆盖写都会 500（content.write / 资源更新 / 部分 add_resource 收尾）
  • 全库 53,003 条记录，scroll 最多只能读到 200 条
  • 相关派生现象：Candidate data is None for label index ... (label: 18446744073709551615) 警告、_strict_scan fail-closed 报错、reindex 报 cursor ended after 4 of 7
  • v0.5.4 → v0.5.4.1 升级未修复

疑点

scroll() 注释写道：

The local engine's scalar sorter does not return records for path/string fields. updated_at is an indexed date-time field on every context collection and provides stable offset pages.

说明作者已知本地引擎 sorter 存在字段限制。实测表明 offset 本身也不被本地引擎的 sorter 尊重，怀疑：

  1. order_by="updated_at" + offset 的组合在本地引擎未实现；
  2. 或 offset 参数在 adapter → native 层丢失（类似 delete_field_data failed, offset %d is invalid when element_size is %lu 那条日志里 %d/%lu 未格式化、参数在 native→logging 桥接时丢失的问题）。

建议修复方向

  • 直接用本地引擎 API 验证 strict_query(order_by=..., offset=N) 是否返回不同页；
  • 若 offset 不可用，改用基于 updated_at 值的 range 游标（updated_at > last_updated_at）而非 offset 分页；
  • 或在 _strict_scan 中对"同页重复"做显式检测，报更明确的错误（当前报错指向"记录缺失"，误导排查方向）。

临时规避

  • 对受影响 URI：删除后只写一次，不要覆盖写；
  • 或写入新路径，旧路径保留只读；
  • 无服务端配置可绕过（_strict_scan 是 fail-closed 保护）。
Dominant language
Python
Stars
41.8k
Forks
3.4k
Avg merge
19h 29m
Merged PRs (30d)
500

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from volcengine/OpenViking

All issues in volcengine/OpenViking

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.