Design production-ready RedisVL index schema migrator
维护者通常 4 天内回复
@nkanu17 已经在做这个了。
开始于 2026年3月19日。
评估
这个 Issue 还没有评估数据。
描述
Summary
Design and propose a production-ready index schema migrator for RedisVL that supports drop-and-recreate migrations, data transformation/backfill, multi-index orchestration, and cluster-safe execution.
Level: Advanced
Current State
- RedisVL supports index create/delete/clear/load flows, but there is no first-class schema migration workflow.
- Schema changes that affect field shape (especially vector datatype/precision changes) can require scanning existing keys and rewriting payload fields before or during reindex.
- Teams currently need ad hoc scripts for:
- dropping/recreating indices,
- transforming indexed documents,
- handling large datasets in Redis Cluster environments.
Problem to Solve
How should RedisVL support production migrations when:
- one index needs multiple sequential schema updates,
- several related indices must migrate together,
- vector fields need precision/type conversion and data rewrite,
- migration must scale with large keyspaces and cluster slot constraints?
Proposed Change
Create a research + design issue that delivers a concrete migration proposal and implementation plan (not full implementation in this task):
-
Migration model
- Define migration spec format (source schema, target schema, transforms, batch size, safety options).
- Support a baseline strategy:
drop -> transform/backfill -> recreate -> validate. - Evaluate optional low-downtime strategy (
shadow index + cutover) and document tradeoffs.
-
Data transformation/backfill
- Define transform hooks for field-level rewrites (e.g., vector
float32 -> float16conversions). - Specify how documents are scanned, transformed, and re-written safely.
- Define transform hooks for field-level rewrites (e.g., vector
-
Multi-index + multi-step orchestration
- Propose dependency-aware orchestration for N indices and ordered updates.
- Include checkpointing/resume semantics for long-running jobs.
-
Production/cluster scalability
- Batch execution and bounded memory guarantees.
- Slot-aware operations and cross-slot-safe deletion/update behavior.
- Throughput controls (concurrency limits, backpressure, retry policy).
-
Safety and observability
- Dry-run mode with migration plan preview and impact estimates.
- Validation checks before/after migration (doc counts, schema checks, sample query checks).
- Structured progress metrics/logging and failure recovery guidance.
Definition of Done
- A design doc (or RFC-style markdown) exists in-repo with:
- architecture and migration lifecycle,
- execution strategies (baseline + optional low-downtime),
- multi-index orchestration approach,
- cluster-scaling strategy,
- failure/rollback recommendations.
- Includes a proposed API surface (Python + optional CLI hooks) and phased implementation plan.
- Includes a breakdown into follow-up implementation issues sized for hackathon teams.
Out of Scope
- Full end-to-end migrator implementation in this issue.
- Guaranteeing zero-downtime for all migration types.
- 主要语言
- Python
- 星标
- 432
- 派生
- 116
- 平均合并
- 5 天 22 小时
- 30 天内合并 PR
- 13
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
redis/redis-vl-python 的其他 Issue
-
from_existing fails on SVS-VAMANA indexes created without compression可能已有人在做 @itsrayeedscodespace 于 2 天前认领。 未关闭bug index
难度 2/5 1-3 小时 新手友好度 84/100
redis/redis-vl-python#787 ·
维护者通常 4 天内回复
-
RedisVL warnings are never shown when the application has not configured logging可能已有人在做 @hippi345 于 2 天前认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 77/100
redis/redis-vl-python#786 ·
维护者通常 4 天内回复
-
exists() depends on exact FT.INFO error wording; an unrecognised "missing index" message makes create() never create the index (regression from #672)可能重新可做 关联的 PR 已关闭且未合并。 未关闭
难度 2/5 1-3 小时 新手友好度 75/100
redis/redis-vl-python#773 · 1 条评论 ·
维护者通常 4 天内回复
-
Vector field with index_missing=True cannot be created可能已有人在做 @RaphaelFakhri 于 12 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
redis/redis-vl-python#770 ·
维护者通常 4 天内回复
-
Add Python 3.15 support可能已有人在做 @vishal-bala 于 14 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 65/100
redis/redis-vl-python#757 · 1 条评论 · 1 个 reaction ·
维护者通常 4 天内回复
查看 redis/redis-vl-python 的全部 Issue
相似的 Issue
-
infrastructure
难度 2/5 1-3 小时 新手友好度 76/100
NatLabRockies/H2Integrate#977 ·
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 72/100
topoteretes/cognee#6470 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
-
area:docs priority:p2 type:docs wave:9
难度 1/5 1 小时以内 新手友好度 84/100
jejjohnson/radtrax#293 ·
维护者通常 1 天内回复
-
bug core plugins
难度 2/5 1-3 小时 新手友好度 68/100
bunkerity/bunkerweb#4121 · 1 条评论 ·
维护者通常 1 天内回复