Hasura metrics, step 2: pool, per-subscription cost, auth hook and middleware subscription limits

Open
  • Is something wrong?
#26,452 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
postgresql, prometheus, typescript
Domain
backend-api-design, observability

Research direction

The work spans the BigBlueButton Hasura fork and bbb-graphql-middleware; the issue does not name source files or tests. Start by reading #26450 and #26451 and locating the existing engine metrics and middleware /userInfo and subscription limit handling. Done means implementing the requested pool, subscription, poller, auth hook and connection limit metrics with bounded labels, plus tests that drive each metric and show its value change.

Written by the indexing model from the issue text.

Description

Context

#26450 (v4.1.x-develop) and #26451 (v3.0.x-develop) switch bbb-graphql-server to Hasura 2.50.7 built from source by the BigBlueButton fork (https://github.com/bigbluebutton/hasura-graphql-engine, release v2.50.7-bbb.1) and expose the engine's built-in metrics on /v1/metrics (Prometheus text format, protected by a secret generated at install time).

This issue tracks the second step: the metrics the engine does not keep today and that matter for operating BigBlueButton, plus the middleware side of the picture. Most items change the fork; the middleware items change bbb-graphql-middleware.

Hasura fork
  1. Postgres connection pool. Connections in use, idle and waiting; time to acquire a connection from the pool; time to open a new connection to Postgres. Every live query poller hits the database every 333 ms and every streaming poller every 150 ms, so pool saturation is the usual way the engine degrades under load.
  2. Pool churn (suggested by @ritzalam). Counters of connections opened, closed and recycled by connection_lifetime. Needed to measure the effect of tuning connection_lifetime (for example 3600 to 300 seconds to reduce Postgres backend memory) from inside the engine.
  3. Per-subscription cost. Operation name as a label on the subscription metrics (cycle time, database execution time, active subscriptions), behind a flag, so the expensive subscriptions can be told apart. Label values are operation names only.
  4. Poller lag (suggested by @ritzalam). hasura_subscription_total_time_seconds (per subscription kind) already gives the duration of each poll cycle, to compare with the refetch interval. Evaluate a counter of cycles that exceeded the interval, so the condition can be alerted on without histogram math.
bbb-graphql-middleware
  1. Auth hook. Latency and failures of /userInfo (the session vars hook called by Hasura on every connection), measured on the middleware side. The engine does not measure its auth webhook.
  2. Subscriptions per connection (suggested by @ritzalam). Concurrent subscriptions per connection as an aggregate (no per-connection label) and a counter of subscriptions rejected at max_connection_concurrent_subscriptions (150). Today #25909 logs the active subscriptions once, when the limit is first hit; a metric shows the build-up before that.
Rules
  • Labels carry operation names and kinds only. No meeting IDs, user IDs or anything else that grows with usage or is close to personal data (point raised by @ritzalam: at fleet scale they multiply series quickly).
  • Metric names and buckets follow the conventions already used by the engine's existing metrics.
  • Every metric comes with a test that drives it and shows the value change.

Out of scope here: OpenTelemetry export (the Collector's Prometheus receiver can already scrape /v1/metrics and ship it over OTLP).

Dominant language
TypeScript
Stars
9.3k
Forks
6k
Avg merge
2d 7h
Merged PRs (30d)
282

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from bigbluebutton/bigbluebutton

All issues in bigbluebutton/bigbluebutton

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.