sync of a lot of small files fails

Abierto
  • ¿Algo no está bien?
#350 15 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
15/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
python
Área
cli, cloud

Línea de trabajo

The payload names only the b2 sync --delete --threads 100 command and two errors from b2_upload: a 408 request_timeout and an IndexError. It does not name the source files that handle the upload or retry path, so start by reproducing with many small files and locating the b2_upload call from the traceback. Done means failed uploads are retried or reported, and the old versions are not deleted for files that did not upload. The thread has been idle for about 8 years, so confirm with maintainers that the issue is still wanted before starting.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

I tried to sync about 1 million small files (Combined size of ~40GB) to backblaze and got a lot of 408 request_timeout errors:

b2_upload(/local/path/to/file, b2/path/to/file, 1499162591000): UnknownError('408 request_timeout Error reading uploaded data: SocketTimeoutException(Read timed out)',) Unknown error: 408 request_timeout Error reading uploaded data: SocketTimeoutException(Read timed out)

And also some other errors:

b2_upload(/local/path/to/file, b2/path/to/file, 1499161555000): IndexError('tuple index out of range',) tuple index out of range

The command I used is:
b2 sync --delete --threads 100 /path/to/local/directory b2://bucketname/foldername

So all of the old file versions were deleted, while all the files that got an error were not uploaded. I would expect that it would retry all the failed uploads.

Lenguaje dominante
Python
Estrellas
638
Forks
138
Merge medio
8 h 27 min
PR fusionados (30 d)
8

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Backblaze/B2_Command_Line_Tool

Todos los issues de Backblaze/B2_Command_Line_Tool

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.