How to calculate metrics? Bucket size, last updated, etc

Aperta
  • Qualcosa non va?
#137 10 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
12/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
python
Ambito
api, cloud

Direzione di ricerca

The payload names no file or test in this repository. The request depends on whether the B2 service exposes per-bucket size, file count and last-upload time, which is a server-side question outside this SDK. Start from the linked B2_Command_Line_Tool issue 404 and the --showSize recursive sum it describes. Done would need a maintainer decision on an API endpoint, so do not start a patch before that is settled.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

We are in the process of setting up monitoring and alerts for our backblaze backups, so we are notified if one of our backup processes stops working.

Some metrics I'd like to track per bucket are:

  • time since last updated (timestamp of most recently uploaded file)
    • this one is the most critical -- is our backup process running at all?
  • total size and number of files
    • we are adding data daily, so in general these numbers should slowly increase over time
  • number of versions
    • to make sure config didn't get accidentally messed up for buckets that should have versions

For total size and number of files -- I found https://github.com/Backblaze/B2_Command_Line_Tool/issues/404 which adds --showSize to the CLI. But I looked at the code that calculates this, and it recursively looks at every file in the bucket and adds it up. That's simply not going to perform well. (It worked fine for a small bucket, but when I tried it for one of our larger buckets I didn't get a result after 15 minutes of waiting and killed the process) What's strange is, I can see this info on the Backblaze website, so it seems like you guys know these stats per bucket? Is there a chance it could be exposed somehow?

For time since last updated, we could add a "canary file" to the root of each bucket, make sure it gets updated regularly, and check that... but it would be far less brittle if backblaze provided this info. Do you store this info? If so, is there any way to access it?

For number of versions, I could parse out lifecycle_rules from the bucket info, so that's fine.

Any guidance here? We use prometheus for metrics, so the plan is to use this python sdk and write a simple client to export the above metrics. It would be generic enough that we could open-source the project so others could use it. But as it stands right now, I can't figure out a way to get enough information to create useful metrics in a generic way.

Thanks!

Lingua principale
Python
Stelle
214
Fork
68
Merge medio
51m
PR unite (30g)
6

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Backblaze/b2-sdk-python

Tutte le issue di Backblaze/b2-sdk-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.