Monitoring: daily usage figures from the access log

Aperta
#2,666 2 commenti 0 reazioni 2 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

@jh-RLI ci sta già lavorando.

Dal 13/10/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

deployment Infrastructure type: enhancement :gear:

Part of #2412. Builds on #2665 (node_exporter). Replaces #2402: the log filter script moves here. Target: Tue 2026-10-13 morning.

Why

First usage figures for the platform - requests, downloads, logins, page views - from the access log of the app container, privacy-friendly (counts only). Visible in Grafana and kept as a daily archive.

What it produces

One script, podman/monitoring/kpi/oep-usage.sh (based on script/logfilter/logfiltering.sh from #2402), with two outputs:

  1. For Grafana: a file oep_usage.prom in node_exporter's textfile format, rewritten every hour for the last 24 hours. Gauges such as oep_usage_requests{class="2xx"}, oep_usage_downloads{format="csv"}, oep_usage_logins, oep_usage_pageviews{area="database"}. Per-table and per-tag figures only as a top 10 - thousands of label values would overload Prometheus.
  2. Archive: one CSV row per calendar day, written shortly after midnight for the day before. Prometheus keeps 30 days; reports need years. The full per-table and per-tag lists go here.

Run by plain systemd user units (not Quadlets - this is a job on the host that reads logs): oep-mon-usage-hourly.service/.timer and oep-mon-usage-daily.service/.timer, installed by podman/monitoring/install.sh. Write both outputs atomically (temporary file, then mv), otherwise node_exporter can read half a file.

Fixes the script needs

  • Lines with - as byte count are not counted. Apache's default log format writes - for an empty response (304, most 204, some 404). The regex expects digits, so these lines are skipped completely: the three lines 200 5120, 304 -, 404 - give total_reqs 1. Accept ([0-9]+|-).
  • It only runs with gawk. The three-argument match() is a gawk extension. With mawk (the default awk on Debian/Ubuntu) it is a syntax error, the output is empty and the script still exits 0. Call gawk explicitly and exit with an error if it is missing.
  • Count status classes 2xx/3xx/4xx/5xx. Today 201 and 204 are in total_reqs but in no status column.
  • Skip the monitoring's own requests: any URL containing probe=blackbox (see #2665).
  • Both table routes. Rows are served under /api/v0/tables/<t>/rows/ and the older /api/v0/schema/<s>/tables/<t>/rows/ (one view, see api/urls.py). The table-download regex only knows the first.
  • Log source. Check how the app container logs on the server: podman inspect oeplatform --format '{{.HostConfig.LogConfig.Type}}'. With journald, podman logs works. With passthrough, podman logs returns nothing and the lines are only in journalctl --user -u oep-oeplatform --since ... --until ... -o cat. Support both and note in the README which one the server uses.
  • Read stdout only (2>&1 mixes Apache's error log into the input).
  • Logins: allauth is mounted at /accounts/ (oeplatform/urls.py). A POST /accounts/login/ is an attempt; count only those answered with 302 (successful).
  • Escape " inside CSV fields.
  • Write outputs under $XDG_STATE_HOME/oep-monitoring/ (default ~/.local/state/oep-monitoring/), never under media/.

Privacy

Write this into the docs page too:

  • Only counts leave the script: per day or per 24 h, plus top-10 table and tag names (both public on the platform). No IP address, no user name, no full URL with query string in any output.
  • The script keeps nothing of the raw log.
  • The log format has no user agent, so crawlers count as visitors. State this as a known limit.

Done when

  • A small test file of log lines - including - byte counts, both table routes, a probe line, a failed and a successful login - gives the expected numbers. Commit it with the expected output under podman/monitoring/kpi/testdata/, plus a one-line check command in the README.
  • systemctl --user list-timers 'oep-mon-usage*' shows both timers; oep_usage.prom updates every hour; the CSV gets one row per day.
  • The figures are queryable in Prometheus (oep_usage_*).
  • Nothing in either output identifies a person.
  • After the PR is open, close #2402 with a link to it.
Lingua principale
Python
Stelle
65
Fork
28
Merge medio
4g 7h
PR unite (30g)
25

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenEnergyPlatform/oeplatform

Tutte le issue di OpenEnergyPlatform/oeplatform

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.