How to calculate metrics? Bucket size, last updated, etc

Abierto
  • ¿Algo no está bien?
#137 10 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
12/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Estancado
Stack tecnológico
python
Área
api, cloud

Línea de trabajo

The payload names no file or test in this repository. The request depends on whether the B2 service exposes per-bucket size, file count and last-upload time, which is a server-side question outside this SDK. Start from the linked B2_Command_Line_Tool issue 404 and the --showSize recursive sum it describes. Done would need a maintainer decision on an API endpoint, so do not start a patch before that is settled.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

We are in the process of setting up monitoring and alerts for our backblaze backups, so we are notified if one of our backup processes stops working.

Some metrics I'd like to track per bucket are:

  • time since last updated (timestamp of most recently uploaded file)
    • this one is the most critical -- is our backup process running at all?
  • total size and number of files
    • we are adding data daily, so in general these numbers should slowly increase over time
  • number of versions
    • to make sure config didn't get accidentally messed up for buckets that should have versions

For total size and number of files -- I found https://github.com/Backblaze/B2_Command_Line_Tool/issues/404 which adds --showSize to the CLI. But I looked at the code that calculates this, and it recursively looks at every file in the bucket and adds it up. That's simply not going to perform well. (It worked fine for a small bucket, but when I tried it for one of our larger buckets I didn't get a result after 15 minutes of waiting and killed the process) What's strange is, I can see this info on the Backblaze website, so it seems like you guys know these stats per bucket? Is there a chance it could be exposed somehow?

For time since last updated, we could add a "canary file" to the root of each bucket, make sure it gets updated regularly, and check that... but it would be far less brittle if backblaze provided this info. Do you store this info? If so, is there any way to access it?

For number of versions, I could parse out lifecycle_rules from the bucket info, so that's fine.

Any guidance here? We use prometheus for metrics, so the plan is to use this python sdk and write a simple client to export the above metrics. It would be generic enough that we could open-source the project so others could use it. But as it stands right now, I can't figure out a way to get enough information to create useful metrics in a generic way.

Thanks!

Lenguaje dominante
Python
Estrellas
214
Forks
68
Merge medio
51 min
PR fusionados (30 d)
6

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Backblaze/b2-sdk-python

Todos los issues de Backblaze/b2-sdk-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.