build.py downloads rebuild inputs without DataCache: no atomic write, truncated files fail checksum until deleted

Abierto
#184 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
65/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
tooling

Línea de trabajo

Start in orfdata/build.py at prepare_sources and RemoteZip, then read the datacache.fetch_file API to understand its verification and retry options. Run the existing tests and inspect how source and derived files are written. Done means an interrupted download leaves no final-path file, a later run succeeds without manual cleanup, and a test covers the interrupted-download case.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

python -m orfdata.build --prepare-sources downloads the uORF rebuild inputs with its own code in orfdata/build.py (prepare_sources and RemoteZip), not through DataCache. ORFdata already depends on DataCache and uses it for every other download.

What the code does on main (cd1b828):

  • Each pinned input is fetched with urllib.request.urlopen and streamed straight into its final path with target.open("wb") (lines 127 and 139). There is no temporary file and no rename, so an interrupted transfer leaves a truncated file under the real name.
  • The SHA-256 pin is checked only after the file has been written (verify(target)). On the next run, if not target.exists() skips the download and verify fails with "Source checksum mismatch". It keeps failing until someone deletes the file by hand.
  • There are no retries, no resume and no size limit. datacache.fetch_file has expected_sha256, expected_size, max_bytes, retries and resume.
  • The benchmark CSV is read from a 4.8 GB figshare ZIP through RemoteZip, a hand-written HTTP-range reader, and then written with write_bytes. That write has the same truncation problem.
  • The derived files (author-hla-mappings.tsv from Rscript and author-hla-counts.json) are also written in place and verified afterwards.

Proposal:

  • Fetch the URL-pinned inputs with datacache.fetch_file(url, destination=..., expected_sha256=..., raw=True). DataCache publishes the file only after it is verified, and it retries and can resume. Sizes could be pinned next to the hashes too.
  • Write each derived file to a temporary path in the same directory, verify it, then rename it into place.
  • Keep RemoteZip only if DataCache can't read a single ZIP member by byte range. Its output then gets the same write, verify and rename treatment.

Acceptance: an interrupted --prepare-sources run leaves no file under a final name, and running it again completes with no manual cleanup. A test fails a download partway through and checks this.

Related: #156 (resume), #157 (reuse pinned sources) and #159 (the viral stack's own urllib code). Found while reviewing #180.

Lenguaje dominante
Python
Estrellas
1
Forks
1
Merge medio
10 h 11 min
PR fusionados (30 d)
51

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de openvax/orfdata

Todos los issues de openvax/orfdata

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.