[PERF] Pre-aggregated per-minute counts (log_counts rollup) for stats, histograms and facets

Aperta
#151 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
mysql, php

Direzione di ricerca

Start by locating AsyncMysqlBatchWriter, the database migrations, the retention code, and the API routing and OpenAPI files. Then trace how entries are inserted and pruned, and how existing CLI commands and tests are structured. Done means implementing the rollup migration, transactional batch updates, backfill command, histogram endpoint and spec, and documented retention behavior, with tests confirming counts against seeded log_entries.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

✨ feature 🎲 database 📊 dashboard 🕓 medium effort 🚀 performance enhancement

Summary

Maintain a small rollup table of counts per minute, app and level, updated as entries are written. Histograms, facet counts and approximate totals can then be read from it instead of scanning log_entries.

Motivation

#56 (metrics/statistics endpoint) and guibranco/logstream-ui#159 (dashboard/overview screen) both need counts over time. With COUNT(*) … GROUP BY on the raw table, the cost grows with the amount of data stored. A rollup table makes these queries tiny, and lets the stats drawer show real totals instead of counting only the entries loaded in the browser.

Implementation

CREATE TABLE IF NOT EXISTS log_counts (
    bucket   DATETIME     NOT NULL,          -- minute, UTC
    app_key  VARCHAR(100) NOT NULL,
    app_id   VARCHAR(100) NOT NULL,
    level    ENUM('debug','info','notice','warning','error','critical') NOT NULL,
    n        INT UNSIGNED NOT NULL,
    PRIMARY KEY (bucket, app_key, app_id, level)
) ENGINE=InnoDB;
  • In AsyncMysqlBatchWriter, group each batch by key in PHP and run one multi-row INSERT … ON DUPLICATE KEY UPDATE n = n + VALUES(n), so there is no extra query per entry.
  • Retention: when rows are pruned, either decrement the counts or keep rollups for longer (they are tiny) and document that they count everything ever ingested. Pick one approach and document it.
  • Backfill: a one-off CLI command that builds log_counts from existing rows.
  • GET /api/stats/histogram?interval=1m|5m|1h&app_key=…&level=…&range=24h reads from the rollup (larger intervals use GROUP BY on the bucket).

Acceptance criteria

  • Migration creates log_counts
  • Batch writer updates counts in the same transaction as the insert
  • Backfill command
  • Histogram endpoint plus OpenAPI spec
  • Retention behaviour decided and documented
  • Tests: counts match COUNT(*) on seeded data

Generated by Claude Code

Lingua principale
PHP
Stelle
1
Fork
0
Merge medio
1h 58m
PR unite (30g)
53

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di guibranco/logstream-server

Tutte le issue di guibranco/logstream-server

Issue simili

Altre issue su PHP

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.