Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

bug(cloud): dashboard read model is rebuilt from full history after every write (~30 s first load)

Open
  • Is something wrong?
#2,200 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, postgresql
Domain
backend, performance

Research direction

Read internal/cloud/cloudstore/cloudstore.go for invalidateDashboardReadModel in WriteChunk and InsertMutationBatch, then follow loadDashboardNeutralReadModel into dashboard_queries.go to understand the full rebuild. Compare incremental updates and serving the last good model during a background rebuild; measure dashboard latency after writes against the reported ~32 seconds. Done means dashboard requests stay interactive as history grows.

Written by the indexing model from the issue text.

Description

status:needs-review type:bug
Existing Issue Search
  • I searched existing issues and will add evidence to a canonical tracker when applicable.

Related: #1832 (measured read-model rebuild cost; measurement only, broader strategies deferred), #1930 / #1931 (OOM and single-flight, fixed — memory is fine here, latency is not).

📝 Observed Consequence

On a small team deployment (6 managed users, autosync on), the first dashboard request after any sync write takes ~30 s. With several active agents a write arrives every few minutes, so most dashboard visits hit a cold rebuild. Browsers and users give up first: the login redirect to /dashboard/ looks like "nothing happens" and nginx logs 499 (client closed) for the first attempts.

🔄 Steps to Reproduce
  1. Run engram cloud serve 3.6.0 with a team-sized history (here: cloud_mutations 421,529 rows / 542 MB, cloud_chunks 2,667 rows / 475 MB, 60 projects).
  2. Let any client push (autosync WriteChunk or InsertMutationBatch); both invalidate the dashboard read model.
  3. Log in at /dashboard/login and follow the redirect to /dashboard/.
✅ Expected Behavior

Dashboard pages stay interactive (a few seconds at most) regardless of write frequency, e.g. by applying new chunks/mutations incrementally to the cached read model, or by serving the last good model while a rebuild runs in the background (stale-while-revalidate), instead of a full history rebuild on the request path.

❌ Actual Behavior

Each write discards the whole permission-neutral read model (invalidateDashboardReadModel in WriteChunk / InsertMutationBatch, internal/cloud/cloudstore/cloudstore.go), and the next request rebuilds it from every chunk and mutation (loadDashboardNeutralReadModel → full build, dashboard_queries.go). Measured:

Request Latency
POST /dashboard/login 0.06 s
first GET /dashboard/ after a write ~32 s (two earlier attempts ended as 499)
same pages once warm 0.03–0.06 s

Rebuild cost grows with history, not with live state: the deployment above has ~56k distinct entities but 421k mutation rows.

Operating System

Linux

Release Version (Report Context)

v3.6.0 (Homebrew binary, engram cloud serve, PostgreSQL 16, systemd, nginx TLS in front)

Agent / Client

Engram Cloud dashboard (browser)

📋 Relevant Logs
nginx access (times are request completion):
07:23:47 "POST /dashboard/login" 303
07:24:10 "GET /dashboard/" 499
07:24:10 "POST /dashboard/login" 303
07:24:22 "GET /dashboard/" 499
07:24:22 "POST /dashboard/login" 303
07:24:54 "GET /dashboard/" 200

engram cloud serve:
07:24:54 dashboard: templ render error: context canceled
07:24:54 dashboard: templ render error: context canceled
💡 Additional Context

#1832 asked whether history-scaling cost stays material enough to justify a broader read-model strategy. This is field evidence that it does for a 6-user team after one week of history. Happy to run a patched build or share more timings.

Dominant language
Go
Stars
7.1k
Forks
729
Avg merge
9h 44m
Merged PRs (30d)
278

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Gentleman-Programming/engram

All issues in Gentleman-Programming/engram

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.