Importing certain databases from zip are extremely slow
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 停滞
- 技术栈
- typescript, vscode
- 领域
- tooling
调研方向
首先跟踪数据库导入路径和当前的流式解压器，然后针对包含大量小文件的归档，对比文件头读取与中央目录查找。完成的标准是大型数据库归档的导入速度显著提升，同时仍能处理文件头不正确的归档。
由索引模型根据 Issue 内容生成。
描述
I can't share the database because it is private, but this database is less than 1GB zipped and about 3.4GB unzipped. Importing this database took over 1.5 hours to completion. I think the problem has to do with the fact that there are over 100,000 source files. First, these files need to be unzipped and placed in a directory. Then we need to re-zip them into a src.zip file. Our current unzipper library uses streaming, but it does read each file into memory separately.
It's possible that the slowness is exacerbated by the fix here: https://github.com/github/vscode-codeql/issues/622. Rather than read the zip in a single pass, we read the zips central directory and then grab each file based on what we find there.
I implemented this fix because some archives do not have correct file headers. The central directory is the source of truth and it exists at the end of the file. This happens when a zip file is updated after it is created.
Most of the time reading the file headers will be correct (except when they aren't). And that is likely faster than the central directory approach, especially when there are lots of small files. So, one possible solution would be to try reading via the file headers first and if that fails, fall back to the central directory.
- 主要语言
- TypeScript
- 星标
- 539
- 派生
- 240
- 平均合并
- 2 天 18 小时
- 30 天内合并 PR
- 37
贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
github/vscode-codeql 的其他 Issue
-
bug VSCode
难度 2/5 1-3 小时 新手友好度 68/100
github/vscode-codeql#4471 · 1 条评论 ·
-
bug enhancement VSCode
难度 2/5 1-3 小时 新手友好度 68/100
github/vscode-codeql#592 · 2 条评论 ·
-
enhancement VSCode
难度 3/5 1-2 天 新手友好度 55/100
github/vscode-codeql#4432 · 1 条评论 ·
-
bug VSCode
难度 3/5 1-2 天 新手友好度 52/100
github/vscode-codeql#4356 ·
-
bug VSCode
难度 3/5 1-2 天 新手友好度 64/100
github/vscode-codeql#4339 · 1 条评论 ·
查看 github/vscode-codeql 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 65/100
-
难度 2/5 1-3 小时 新手友好度 75/100
-
bug v2
难度 2/5 1-3 小时 新手友好度 75/100
modelcontextprotocol/inspector#2458 · 1 条评论 ·
-
难度 1/5 1 小时以内 新手友好度 75/100
railmapgen/rmp-gallery#4068 ·
-
Mend: dependency security vulnerability status: needs triage 🕵️♀️
难度 2/5 1-3 小时 新手友好度 70/100
carbon-design-system/ibm-products#9907 ·