Importing certain databases from zip are extremely slow
還沒有人認領這個 Issue。
評估
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 新手友好度
- 35/100
- Issue 類型
- 缺陷
- 描述清晰度
- 基本清楚
- 活躍度
- 停滯
- 技術堆疊
- typescript, vscode
- 領域
- tooling
研究方向
首先追蹤資料庫匯入路徑和目前的串流解壓縮器，接著針對包含許多小檔案的封存檔，比較檔案標頭讀取與中央目錄查找。完成的標準是大型資料庫封存檔的匯入速度大幅提升，同時仍能處理檔案標頭不正確的封存檔。
由索引模型根據 Issue 內容生成。
描述
I can't share the database because it is private, but this database is less than 1GB zipped and about 3.4GB unzipped. Importing this database took over 1.5 hours to completion. I think the problem has to do with the fact that there are over 100,000 source files. First, these files need to be unzipped and placed in a directory. Then we need to re-zip them into a src.zip file. Our current unzipper library uses streaming, but it does read each file into memory separately.
It's possible that the slowness is exacerbated by the fix here: https://github.com/github/vscode-codeql/issues/622. Rather than read the zip in a single pass, we read the zips central directory and then grab each file based on what we find there.
I implemented this fix because some archives do not have correct file headers. The central directory is the source of truth and it exists at the end of the file. This happens when a zip file is updated after it is created.
Most of the time reading the file headers will be correct (except when they aren't). And that is likely faster than the central directory approach, especially when there are lots of small files. So, one possible solution would be to try reading via the file headers first and if that fails, fall back to the central directory.
- 主要語言
- TypeScript
- 星號
- 539
- 分支
- 240
- 平均合併
- 2 天 18 小時
- 30 天內合併 PR
- 37
貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
github/vscode-codeql 的其他 Issue
-
bug VSCode
難度 2/5 1-3 小時 新手友好度 68/100
github/vscode-codeql#4471 · 1 則留言 ·
-
bug enhancement VSCode
難度 2/5 1-3 小時 新手友好度 68/100
github/vscode-codeql#592 · 2 則留言 ·
-
enhancement VSCode
難度 3/5 1-2 天 新手友好度 55/100
github/vscode-codeql#4432 · 1 則留言 ·
-
bug VSCode
難度 3/5 1-2 天 新手友好度 52/100
github/vscode-codeql#4356 ·
-
bug VSCode
難度 3/5 1-2 天 新手友好度 64/100
github/vscode-codeql#4339 · 1 則留言 ·
查看 github/vscode-codeql 的全部 Issue
相似的 Issue
-
blocklist removal
難度 2/5 1-3 小時 新手友好度 65/100
MetaMask/eth-phishing-detect#296544 ·
-
難度 2/5 1-3 小時 新手友好度 70/100
pastelsky/bundlephobia#1122 ·
-
難度 2/5 1-3 小時 新手友好度 70/100
-
category/development priority/P2 scope/file-operations scope/testing type/enhancement
難度 2/5 1-3 小時 新手友好度 75/100
-
難度 2/5 1-3 小時 新手友好度 75/100