upload: nonexistent paths are silently ignored

Abierto
  • ¿Algo no está bien?
#1,974 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
45/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
cli, data

Línea de trabajo

Start with _partition_upload_assets() and under_paths() in dandi/upload.py, where a missing path matches nothing, then report_omitted_paths(). Check how path_is_subpath() handles a path that equals a prefix, and the commented-out click.Path(exists=True) in cmd_upload. Done means a missing path without --sync fails and names it (broken symlinks still accepted), and with --sync a path matching nothing on the server gets a warning. The --sync deletion question needs a maintainer decision first.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

UX

dandi upload says nothing when a requested path does not exist locally. With a typo such as dandi upload sub-1/ (meant sub-01/), nothing reports the path. Reading upload(), the run then goes ahead with only dandiset.yaml.

Found while reviewing #1932, which makes find_dandi_files() raise FileNotFoundError for missing paths (#1094). That PR does not reach upload, because upload() never passes the requested paths to find_dandi_files(). It walks the whole Dandiset with dandiset.assets(allow_all=True), then filters with under_paths() in _partition_upload_assets() (dandi/upload.py). A missing path matches nothing:

>>> _partition_upload_assets(dandiset.assets(allow_all=True), [PurePosixPath("missing.nwb")], False)
([], [])

Nothing is selected and nothing is omitted, so report_omitted_paths() has nothing to warn about either. The click.Path(exists=True) on the paths argument of cmd_upload is commented out.

--sync rules out a blanket error

A missing path can be intentional with --sync. After rm -r sub-01, dandi upload --sync sub-01 offers to delete the remote assets under sub-01/. With --sync, a missing path currently gives:

Missing local path On the server Result
directory, e.g. sub-01 assets under sub-01/ offered for deletion (expected)
file, e.g. sub-01/a.nwb sub-01/a.nwb not deleted, because path_is_subpath() returns False when the path equals the prefix
typo, e.g. sub-1 nothing nothing happens, no message
Possible behavior
  • Without --sync: fail before doing anything and name the missing paths, as dandi validate already does with LinkAwarePath(lexists=True). Broken symlinks, i.e. unfetched DataLad/git-annex content, should still be accepted.
  • With --sync: accept a missing path only if it matches something on the server, and warn or error otherwise. Separately, decide whether --sync should delete a remote file when that file's own path was given and is gone locally.
Lenguaje dominante
Python
Estrellas
28
Forks
37
Merge medio
3 d 21 h
PR fusionados (30 d)
13

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dandi/dandi-cli

Todos los issues de dandi/dandi-cli

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.