Bug: running pull (or update --resume via CLI) doesn't finish incomplete youtube-dl download

Open
#52 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
backend, cli

Research direction

Start by reproducing the reported archivebox update --resume <timestamp> flow after cancelling a YouTube download, then trace the youtube-dl handling used by the CLI and web UI. Done means an incomplete download is detected and re-run so its partial files are compiled into the final webm, with the update no longer reporting a false success.

Written by the indexing model from the issue text.

Description

Describe the bug

A YouTube video didn't complete its download before the timeout, so I have lots of files in the site's media directory representing subtitles, partial video files etc. but youtube-dl hasn't finished compiling them into one webm file. However, selecting the site and clicking "pull" in the web UI or running archivebox update --resume <timestamp> doesn't appear to re-run youtube-dl to finish the video archival.

Steps to reproduce
  1. Start archival of a YouTube video.
  2. Cancel the youtube-dl step so that the download is only partial.
  3. Attempt to resume download with web UI or CLI and note that it doesn't complete the video archival.
Screenshots or log output

Partially redacted output, showing "success" for the video in question despite it not finishing the archival.

$ docker compose run -e SAVE_MEDIA=1 archivebox update --resume 1673798553.228281
[i] [2023-02-19 16:45:22] ArchiveBox v0.6.2: archivebox update --resume 1673798553.228281
    > /data


[▶] [2023-02-19 16:46:54] Starting archiving of 14 snapshots in index...

[redacted]

[√] [2023-02-19 16:47:00] "<redacted>"
    https://www.youtube.com/watch?v=<redacted>
    √ ./archive/1673798553.228281
        268 files (310.7 MB) in 0:00:00s 

[redacted]

[√] [2023-02-19 16:47:03] Update of 14 pages complete (9.03 sec)
    - 12 links skipped
    - 2 links updated
ArchiveBox version
ArchiveBox v0.6.2
Cpython Linux Linux-5.4.0-139-generic-x86_64-with-glibc2.28 x86_64
IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY     v0.6.2          valid     /usr/local/bin/archivebox                                                   
 √  PYTHON_BINARY         v3.9.5          valid     /usr/local/bin/python3.9                                                    
 √  DJANGO_BINARY         v3.1.10         valid     /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py           
 √  CURL_BINARY           v7.64.0         valid     /usr/bin/curl                                                               
 √  WGET_BINARY           v1.20.1         valid     /usr/bin/wget                                                               
 √  NODE_BINARY           v15.14.0        valid     /usr/bin/node                                                               
 √  SINGLEFILE_BINARY     v0.3.16         valid     /node/node_modules/single-file/cli/single-file                              
 √  READABILITY_BINARY    v0.0.2          valid     /node/node_modules/readability-extractor/readability-extractor              
 √  MERCURY_BINARY        v1.0.0          valid     /node/node_modules/@postlight/mercury-parser/cli.js                         
 -  GIT_BINARY            -               disabled  /usr/bin/git                                                                
 √  YOUTUBEDL_BINARY      v2021.04.26     valid     /usr/local/bin/youtube-dl                                                   
 √  CHROME_BINARY         v90.0.4430.93   valid     /usr/bin/chromium                                                           
 √  RIPGREP_BINARY        v0.10.0         valid     /usr/bin/rg                                                                 

[i] Source-code locations:
 √  PACKAGE_DIR           22 files        valid     /app/archivebox                                                             
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates                                                   
 -  CUSTOM_TEMPLATES_DIR  -               disabled                                                                              

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled                                                                              
 -  COOKIES_FILE          -               disabled                                                                              

[i] Data locations:
 √  OUTPUT_DIR            5 files         valid     /data                                                                       
 √  SOURCES_DIR           71 files        valid     ./sources                                                                   
 √  LOGS_DIR              1 files         valid     ./logs                                                                      
 √  ARCHIVE_DIR           1258 files      valid     ./archive                                                                   
 √  CONFIG_FILE           81.0 Bytes      valid     ./ArchiveBox.conf                                                           
 √  SQL_INDEX             9.6 MB          valid     ./index.sqlite3
Dominant language
Python
Stars
14
Forks
3
Avg merge
3h 6m
Merged PRs (30d)
17

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ArchiveBox/abx-plugins

All issues in ArchiveBox/abx-plugins

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.