Wrong content-length header for datasets
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 32/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- data
Research direction
Start by running the listed wget command against the dataset URL and compare the Content-Length header with the bytes actually served. The issue is done when the header matches the downloaded file size; the payload does not identify a repository file or entry point to change.
Written by the indexing model from the issue text.
Description
Describe the bug
When downloading the datasets from https://x.com/i/communitynotes/download-data using wget, it hangs, not receiving more data, because the content-length header is too big (566M) for the file being served (185M).
To Reproduce
wget https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
wget https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
--2024-09-07 13:43:04-- https://ton.twimg.com/birdwatch-public-data/2024/09/07/notes/notes-00000.tsv
Resolving ton.twimg.com (ton.twimg.com)... 2606:2800:233:7ee2:97c:ab4c:6c70:be36, 152.199.21.140
Connecting to ton.twimg.com (ton.twimg.com)|2606:2800:233:7ee2:97c:ab4c:6c70:be36|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 593120688 (566M) [text/tab-separated-values]
Saving to: ‘notes-00000.tsv’
notes-00000.tsv 32%[=======> ] 185.19M --.-KB/s eta 2m 58s
Expected behavior
The content-length header should be set to the file size.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 350
- Avg merge
- 11m
- Merged PRs (30d)
- 1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from twitter/communitynotes
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
twitter/communitynotes#463 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
twitter/communitynotes#440 ·
-
Difficulty 5/5 Over a week Newbie friendliness 10/100
twitter/communitynotes#471 · 1 comment ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
twitter/communitynotes#469 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
twitter/communitynotes#460 ·
All issues in twitter/communitynotes
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100