Monitoring: daily usage figures from the access log
I maintainer di solito rispondono entro 3 giorni
@jh-RLI ci sta già lavorando.
Dal 13/10/2026.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
Part of #2412. Builds on #2665 (node_exporter). Replaces #2402: the log filter script moves here. Target: Tue 2026-10-13 morning.
Why
First usage figures for the platform - requests, downloads, logins, page views - from the access log of the app container, privacy-friendly (counts only). Visible in Grafana and kept as a daily archive.
What it produces
One script, podman/monitoring/kpi/oep-usage.sh (based on script/logfilter/logfiltering.sh from #2402), with two outputs:
- For Grafana: a file
oep_usage.promin node_exporter's textfile format, rewritten every hour for the last 24 hours. Gauges such asoep_usage_requests{class="2xx"},oep_usage_downloads{format="csv"},oep_usage_logins,oep_usage_pageviews{area="database"}. Per-table and per-tag figures only as a top 10 - thousands of label values would overload Prometheus. - Archive: one CSV row per calendar day, written shortly after midnight for the day before. Prometheus keeps 30 days; reports need years. The full per-table and per-tag lists go here.
Run by plain systemd user units (not Quadlets - this is a job on the host that reads logs): oep-mon-usage-hourly.service/.timer and oep-mon-usage-daily.service/.timer, installed by podman/monitoring/install.sh. Write both outputs atomically (temporary file, then mv), otherwise node_exporter can read half a file.
Fixes the script needs
- Lines with
-as byte count are not counted. Apache's default log format writes-for an empty response (304, most 204, some 404). The regex expects digits, so these lines are skipped completely: the three lines200 5120,304 -,404 -givetotal_reqs 1. Accept([0-9]+|-). - It only runs with gawk. The three-argument
match()is a gawk extension. Withmawk(the defaultawkon Debian/Ubuntu) it is a syntax error, the output is empty and the script still exits 0. Callgawkexplicitly and exit with an error if it is missing. - Count status classes
2xx/3xx/4xx/5xx. Today 201 and 204 are intotal_reqsbut in no status column. - Skip the monitoring's own requests: any URL containing
probe=blackbox(see #2665). - Both table routes. Rows are served under
/api/v0/tables/<t>/rows/and the older/api/v0/schema/<s>/tables/<t>/rows/(one view, seeapi/urls.py). The table-download regex only knows the first. - Log source. Check how the app container logs on the server:
podman inspect oeplatform --format '{{.HostConfig.LogConfig.Type}}'. Withjournald,podman logsworks. Withpassthrough,podman logsreturns nothing and the lines are only injournalctl --user -u oep-oeplatform --since ... --until ... -o cat. Support both and note in the README which one the server uses. - Read stdout only (
2>&1mixes Apache's error log into the input). - Logins: allauth is mounted at
/accounts/(oeplatform/urls.py). APOST /accounts/login/is an attempt; count only those answered with302(successful). - Escape
"inside CSV fields. - Write outputs under
$XDG_STATE_HOME/oep-monitoring/(default~/.local/state/oep-monitoring/), never undermedia/.
Privacy
Write this into the docs page too:
- Only counts leave the script: per day or per 24 h, plus top-10 table and tag names (both public on the platform). No IP address, no user name, no full URL with query string in any output.
- The script keeps nothing of the raw log.
- The log format has no user agent, so crawlers count as visitors. State this as a known limit.
Done when
- A small test file of log lines - including
-byte counts, both table routes, a probe line, a failed and a successful login - gives the expected numbers. Commit it with the expected output underpodman/monitoring/kpi/testdata/, plus a one-line check command in the README. systemctl --user list-timers 'oep-mon-usage*'shows both timers;oep_usage.promupdates every hour; the CSV gets one row per day.- The figures are queryable in Prometheus (
oep_usage_*). - Nothing in either output identifies a person.
- After the PR is open, close #2402 with a link to it.
- Lingua principale
- Python
- Stelle
- 65
- Fork
- 28
- Merge medio
- 4g 7h
- PR unite (30g)
- 25
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di OpenEnergyPlatform/oeplatform
-
API bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
OpenEnergyPlatform/oeplatform#2580 ·
I maintainer di solito rispondono entro 3 giorni
-
Redesign the database page to support dataset creationForse già presa @bmlancien l’ha presa 4 giorni fa. Apertadata-upload
OpenEnergyPlatform/oeplatform#2762 · 1 commento · 3 assegnatari ·
I maintainer di solito rispondono entro 3 giorni
-
Improve navigation links in Core Features and Integrated WorkflowsForse già presa @tomi-rli l’ha presa 19 giorni fa. ApertaHomepage
OpenEnergyPlatform/oeplatform#2746 · 1 assegnatario ·
I maintainer di solito rispondono entro 3 giorni
-
Enhance the new OEP homepageForse già presa @jh-RLI l’ha presa 19 giorni fa. ApertaHomepage
OpenEnergyPlatform/oeplatform#2745 · 2 assegnatari ·
I maintainer di solito rispondono entro 3 giorni
-
Display child nodes in the OEO Viewer Graph ViewForse già presa @jh-RLI l’ha presa 23 giorni fa. Apertaoeo viewer
OpenEnergyPlatform/oeplatform#2739 · 1 assegnatario ·
I maintainer di solito rispondono entro 3 giorni
Tutte le issue di OpenEnergyPlatform/oeplatform
Issue simili
-
needs-triage
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 88/100
flashinfer-ai/flashinfer#9803 ·
I maintainer di solito rispondono entro 1 giorno
-
bug frontend good first issue help wanted
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
oscarbol09/thesisforge#61 ·
-
api bug performance
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
oscarbol09/audiobard#159 ·
-
area:web p2 product:business triage:active type:bug
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 89/100
cuadrao-ai/argus#2328 ·
I maintainer di solito rispondono entro 1 giorno
-
bug frontend good first issue help wanted
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
nikhilb2/simple_invoicing#907 ·