Bug: running pull (or update --resume via CLI) doesn't finish incomplete youtube-dl download
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
Research direction
Start by reproducing the reported archivebox update --resume <timestamp> flow after cancelling a YouTube download, then trace the youtube-dl handling used by the CLI and web UI. Done means an incomplete download is detected and re-run so its partial files are compiled into the final webm, with the update no longer reporting a false success.
Written by the indexing model from the issue text.
Description
Describe the bug
A YouTube video didn't complete its download before the timeout, so I have lots of files in the site's media directory representing subtitles, partial video files etc. but youtube-dl hasn't finished compiling them into one webm file. However, selecting the site and clicking "pull" in the web UI or running archivebox update --resume <timestamp> doesn't appear to re-run youtube-dl to finish the video archival.
Steps to reproduce
- Start archival of a YouTube video.
- Cancel the youtube-dl step so that the download is only partial.
- Attempt to resume download with web UI or CLI and note that it doesn't complete the video archival.
Screenshots or log output
Partially redacted output, showing "success" for the video in question despite it not finishing the archival.
$ docker compose run -e SAVE_MEDIA=1 archivebox update --resume 1673798553.228281
[i] [2023-02-19 16:45:22] ArchiveBox v0.6.2: archivebox update --resume 1673798553.228281
> /data
[▶] [2023-02-19 16:46:54] Starting archiving of 14 snapshots in index...
[redacted]
[√] [2023-02-19 16:47:00] "<redacted>"
https://www.youtube.com/watch?v=<redacted>
√ ./archive/1673798553.228281
268 files (310.7 MB) in 0:00:00s
[redacted]
[√] [2023-02-19 16:47:03] Update of 14 pages complete (9.03 sec)
- 12 links skipped
- 2 links updated
ArchiveBox version
ArchiveBox v0.6.2
Cpython Linux Linux-5.4.0-139-generic-x86_64-with-glibc2.28 x86_64
IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep
[i] Dependency versions:
√ ARCHIVEBOX_BINARY v0.6.2 valid /usr/local/bin/archivebox
√ PYTHON_BINARY v3.9.5 valid /usr/local/bin/python3.9
√ DJANGO_BINARY v3.1.10 valid /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py
√ CURL_BINARY v7.64.0 valid /usr/bin/curl
√ WGET_BINARY v1.20.1 valid /usr/bin/wget
√ NODE_BINARY v15.14.0 valid /usr/bin/node
√ SINGLEFILE_BINARY v0.3.16 valid /node/node_modules/single-file/cli/single-file
√ READABILITY_BINARY v0.0.2 valid /node/node_modules/readability-extractor/readability-extractor
√ MERCURY_BINARY v1.0.0 valid /node/node_modules/@postlight/mercury-parser/cli.js
- GIT_BINARY - disabled /usr/bin/git
√ YOUTUBEDL_BINARY v2021.04.26 valid /usr/local/bin/youtube-dl
√ CHROME_BINARY v90.0.4430.93 valid /usr/bin/chromium
√ RIPGREP_BINARY v0.10.0 valid /usr/bin/rg
[i] Source-code locations:
√ PACKAGE_DIR 22 files valid /app/archivebox
√ TEMPLATES_DIR 3 files valid /app/archivebox/templates
- CUSTOM_TEMPLATES_DIR - disabled
[i] Secrets locations:
- CHROME_USER_DATA_DIR - disabled
- COOKIES_FILE - disabled
[i] Data locations:
√ OUTPUT_DIR 5 files valid /data
√ SOURCES_DIR 71 files valid ./sources
√ LOGS_DIR 1 files valid ./logs
√ ARCHIVE_DIR 1258 files valid ./archive
√ CONFIG_FILE 81.0 Bytes valid ./ArchiveBox.conf
√ SQL_INDEX 9.6 MB valid ./index.sqlite3
- Dominant language
- Python
- Stars
- 14
- Forks
- 3
- Avg merge
- 3h 6m
- Merged PRs (30d)
- 17
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ArchiveBox/abx-plugins
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
ArchiveBox/abx-plugins#55 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
ArchiveBox/abx-plugins#37 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
ArchiveBox/abx-plugins#36 · 3 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
ArchiveBox/abx-plugins#33 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
ArchiveBox/abx-plugins#32 ·
All issues in ArchiveBox/abx-plugins
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100