Speed-up database downloads
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- github, typescript, vscode
- Domain
- developer-experience, devtools
Research direction
Start with the provided bash script and its GitHub API endpoint, /repos/{owner}/{repo}/code-scanning/codeql/databases/{language}, and run the single- and multi-chunk examples. Trace how the extension currently downloads databases, then define completion as safely issuing concurrent range requests and reconstructing the original database archive.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
I feel like it takes too long to download a CodeQL database from GitHub into VSCode.
Describe the solution you'd like
Use multi-threaded downloads to speed things up.
Describe alternatives you've considered
N/A
Additional context
For example the QL database from github/codeql is only 160MB, but it takes 2 minutes to download. If I concurrently download 10 chunks of the file the download takes less than 10 seconds. I wrote a small bash script to demonstrate.
A single 160MB chunk:
time sh script.sh github/codeql ql 1
gh api -H Accept: application/zip -H Range: bytes=0-165712932 /repos/github/codeql/code-scanning/codeql/databases/ql
real 2m9.894s
user 0m0.439s
sys 0m1.426s
and a download with 10 chunks of 16MB:
time sh script.sh github/codeql ql 10
gh api -H Accept: application/zip -H Range: bytes=0-16571293 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=16571294-33142587 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=33142588-49713881 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=49713882-66285175 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=66285176-82856469 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=82856470-99427763 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=99427764-115999057 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=115999058-132570351 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=132570352-149141645 /repos/github/codeql/code-scanning/codeql/databases/ql
gh api -H Accept: application/zip -H Range: bytes=149141646-165712932 /repos/github/codeql/code-scanning/codeql/databases/ql
real 0m9.752s
user 0m1.069s
sys 0m2.009s
The script
#! /bin/bash
nwo="$1"
lang="$2"
count="$3"
URL="/repos/${nwo}/code-scanning/codeql/databases/${lang}"
SIZE=$(gh api -H "Accept: application/zip" -H "Range: bytes=0-1" -i "${URL}" | tr -d '\r' | grep "Content-Range: bytes 0-1/" | cut -d / -f 2)
CHUNK_SIZE=$(expr "${SIZE}" / "${count}")
start=0
parts=""
for i in $(seq $(expr "${count}" - 1))
do
end=$(expr "${start}" + "${CHUNK_SIZE}")
echo gh api -H "Accept: application/zip" -H "Range: bytes=${start}-${end}" "${URL}"
gh api -H "Accept: application/zip" -H "Range: bytes=${start}-${end}" "${URL}" > "part-$i" &
start=$(expr "${end}" + 1)
parts="${parts}part-${i} "
done
if [ "${start}" -lt "${SIZE}" ] ; then
echo gh api -H "Accept: application/zip" -H "Range: bytes=${start}-${SIZE}" "${URL}"
gh api -H "Accept: application/zip" -H "Range: bytes=${start}-${SIZE}" "${URL}" > "part-${count}"
parts="${parts}part-${count}"
fi
wait
cat $parts > database.zip
rm -f $parts
- Dominant language
- TypeScript
- Stars
- 539
- Forks
- 240
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 29
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from github/vscode-codeql
-
bug VSCode
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
github/vscode-codeql#4471 · 1 comment ·
-
bug enhancement VSCode
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
github/vscode-codeql#592 · 2 comments ·
-
enhancement VSCode
Difficulty 3/5 1-2 days Newbie friendliness 55/100
github/vscode-codeql#4432 · 1 comment ·
-
bug VSCode
Difficulty 3/5 1-2 days Newbie friendliness 52/100
github/vscode-codeql#4356 ·
-
bug VSCode
Difficulty 3/5 1-2 days Newbie friendliness 64/100
github/vscode-codeql#4339 · 1 comment ·
All issues in github/vscode-codeql
Similar issues
-
clawsweeper:linked-pr-open clawsweeper:no-new-fix-pr clawsweeper:source-repro impact:message-loss issue-rating: 🦞 diamond lobster maturity:stable P2
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#616 ·
-
ZCode 3.14.3 に対応する Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
supermomonga/zcode-acp#24 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
growthbook/growthbook#7100 ·
-
triage
Difficulty 1/5 1-3 hours Newbie friendliness 88/100