[Question] Explain how sync between buckets work; slow overall speed

Abierto
  • ¿Algo no está bien?
#917 8 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
1-2 días
Aptitud para principiantes
12/100
Tipo de issue
Documentación
Claridad
Necesita aclaración
Estado de actividad
Estancado
Stack tecnológico
python
Área
cloud, documentation

Línea de trabajo

The payload names no source files, tests or entry points, only the b2 sync command with its --threads, --delete and --replaceNewer flags. Start by reading the sync command's help output and the project docs to see what they say about how bucket-to-bucket sync works. Done would mean a written explanation of the sync process and its throughput limits, which requires maintainer input on the design.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

I have a sourceBucket with 30.000 images (50-100 KB each), roughly 1.4 GB storage in total, and I want to "sync" it from to a destinationBucket.

b2 sync --threads <10|100|500> --delete --replaceNewer b2://sourceBucket b2://destinationBucket

As this concern many files, or even a large filesize, I'm stumped that it takes 1.5 to 2 hours to complete. The output of the command above shows between 70-100 kB/s (deteriorating over time), which seems kind of low, even when I run this on a production server with 10 Gbps up/down.

Question

As it seems the network speed, or the amount of threads, don't really have much impact on the overall performance, so could you provide me answers on the following questions:

  1. How does sync between buckets work?
    • I cannot find in the documentation on how this works.
    • Does my environment e.g. network speed, system hardware, or other requirements have any influence on syncs between buckets.
  2. Why does it take so long to for a relatively small amount of files, of such a small size, to be synced to another bucket?
  3. What can be improved on my side to sync it faster?
    • I've already tried tweaking the threads argument, but it only differs in magnitude of +/- 10 kB/s.
Lenguaje dominante
Python
Estrellas
638
Forks
138
Merge medio
8 h 27 min
PR fusionados (30 d)
8

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Backblaze/B2_Command_Line_Tool

Todos los issues de Backblaze/B2_Command_Line_Tool

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.