Index is apparently limited to 4 GB · hound-search/hound#351

(7 评论) (0 反应) (0 负责人)JavaScript (5,470 star) (606 fork)batch import

enhancementhelp wanted

描述

:wave: Hound developers!

I am trying to index a pretty large repo (144GB - all current sources of openSUSE), and unsurprisingly the index turns out to be larger than 4GB, thus I hit this fatal message:

https://github.com/hound-search/hound/blob/e3b1b43eb872e47af1de1b0e15d0ec3ac5c51dc4/codesearch/index/write.go#L561

Would it be possible / how hard would it be to support larger indexes?

I only had a brief look at read.go and it seems to me that 32 bit offsets are part of the index file format, so changing that would require re-indexing/converting/supporting two file formats, is that correct?

Thanks for all your efforts on Hound!

贡献者指南

技术栈: go
领域: backendsearch
议题类型: feature
难度: 4
预计时间: over 1 week
活动状态: stale
清晰度: clear
前置要求: Go programming languageunderstanding of binary file formatsfamiliarity with Hound's index packageexperience with backward compatibility
新手友好度: 15
研究方向: The issue is about supporting indexes larger than 4GB by changing the offset size from 32 bit to 64 bit in the index file format. Key files are codesearch/index/write.go (line 561) and read.go. The challenge is backward compatibility; consider supporting two file formats or a migration path. Review the 7 comments in the issue for any suggested approaches or known constraints. No linked PRs exist, so the solution may require designing a new version of the index format.