How to calculate metrics? Bucket size, last updated, etc

Offen
  • Stimmt etwas nicht?
#137 10 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
12/100
Issue-Typ
Feature
Klarheit
Muss geklärt werden
Aktivitätsstatus
Veraltet
Tech-Stack
python
Bereich
api, cloud

Rechercherichtung

The payload names no file or test in this repository. The request depends on whether the B2 service exposes per-bucket size, file count and last-upload time, which is a server-side question outside this SDK. Start from the linked B2_Command_Line_Tool issue 404 and the --showSize recursive sum it describes. Done would need a maintainer decision on an API endpoint, so do not start a patch before that is settled.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

enhancement

We are in the process of setting up monitoring and alerts for our backblaze backups, so we are notified if one of our backup processes stops working.

Some metrics I'd like to track per bucket are:

  • time since last updated (timestamp of most recently uploaded file)
    • this one is the most critical -- is our backup process running at all?
  • total size and number of files
    • we are adding data daily, so in general these numbers should slowly increase over time
  • number of versions
    • to make sure config didn't get accidentally messed up for buckets that should have versions

For total size and number of files -- I found https://github.com/Backblaze/B2_Command_Line_Tool/issues/404 which adds --showSize to the CLI. But I looked at the code that calculates this, and it recursively looks at every file in the bucket and adds it up. That's simply not going to perform well. (It worked fine for a small bucket, but when I tried it for one of our larger buckets I didn't get a result after 15 minutes of waiting and killed the process) What's strange is, I can see this info on the Backblaze website, so it seems like you guys know these stats per bucket? Is there a chance it could be exposed somehow?

For time since last updated, we could add a "canary file" to the root of each bucket, make sure it gets updated regularly, and check that... but it would be far less brittle if backblaze provided this info. Do you store this info? If so, is there any way to access it?

For number of versions, I could parse out lifecycle_rules from the bucket info, so that's fine.

Any guidance here? We use prometheus for metrics, so the plan is to use this python sdk and write a simple client to export the above metrics. It would be generic enough that we could open-source the project so others could use it. But as it stands right now, I can't figure out a way to get enough information to create useful metrics in a generic way.

Thanks!

Vorherrschende Sprache
Python
Sterne
214
Forks
68
Ø Merge
51 Min.
Gemergte PRs (30 T.)
6

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus Backblaze/b2-sdk-python

Alle Issues in Backblaze/b2-sdk-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.