replication.databases: "*" subscribes databases with no replicable tables, causing an endless subscribe/timeout/close loop (~4,300/node/day)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- node.js, typescript
- Domain
- databases
Research direction
Start in replication/replicator.ts at forReplicatedDatabase and inspect how the wildcard/default predicate uses hasExplicitlyReplicatedTable(). Verify the change with a local database containing no replicable tables, while preserving the !database bootstrap path; done means that database is not subscribed under wildcard configuration and the retry loop stops.
Written by the indexing model from the issue text.
Description
Suggested priority: P3 — pure noise/waste, no data at risk, and the fix is a one-line
predicate change. Prioritized at all because ~44% of a quiet node's error-level lines are this
loop, which materially slowed diagnosis of the P0/P1 replication issues (#683, #684).
Area: replication · Line refs: v5.2.1 · Seen on: 5.2.1 (4-node production cluster)
Problem
With replication.databases: "*", every local database is subscribed for replication — including a
database that has no replicable tables at all. The peer correctly reports that it has nothing
registered for that database, subscription setup never completes, and #642's setup watchdog closes
the socket so the subscriber can retry. The retry hits the same wall, so the cycle repeats forever.
Measured on an idle-ish production cluster, per node, over one hour:
| signal | count/hour |
|---|---|
Timed out waiting for database subscription setup for <db> |
170–173 |
No database named "<db>" was declared and registered |
148–149 |
Disconnected from wss://<peer>:9933 (db: "<db>") |
181–188 |
That is ~4,300 connect/timeout/close cycles per node per day, indefinitely. On a node not otherwise
busy, these were 173 of 393 total [error] lines (44%).
The database in question is an application database whose single table is declared
@table(database: "coordination", replicate: false) — deliberately node-local (it backs a per-node
SharedArrayBuffer). Nothing about it should ever be replicated, and the application is not doing
anything unusual: it simply exists as a database, and "*" means "subscribe to everything".
Cause
replication/replicator.ts, forReplicatedDatabase:
if (
options?.databases === undefined ||
options.databases === '*' ||
options.databases.includes(databaseName) ||
options.databases.some?.((dbConfig) => dbConfig.name === databaseName) ||
!database
)
callback(database, databaseName, true);
else if (hasExplicitlyReplicatedTable(databaseName)) callback(database, databaseName, false);
The predicate that would exclude this database already exists — hasExplicitlyReplicatedTable() —
but it is only consulted on the else branch. The wildcard/default path subscribes unconditionally.
Suggested fix
Consult hasExplicitlyReplicatedTable(databaseName) on the wildcard/default path too: a database
with zero replicable tables should not be subscribed regardless of how replication.databases is
configured. (An existing local database with no replicated tables is knowable locally, so this
needs no protocol change. Keep the !database case as-is — that is the isLeader/bootstrap path for
databases that do not exist yet.)
Alternatively/additionally, treat "peer reports the database is not registered" as a terminal
condition for that (peer, database) subscription rather than a retryable one, with backoff instead
of an unbounded tight retry.
Why it matters beyond waste
The volume of error-level output actively hinders diagnosis. While investigating #683 on this
cluster, I initially attributed two page_cache reconnects to this churn purely from temporal
proximity; the counts refuted it (2,090 coordination disconnects vs 2 page_cache disconnects over
the same 12h). Filling error logs at ~4,300/day/node with a condition that is both expected and
permanent makes real replication faults materially harder to find.
Workaround, and why it is not a good one
Setting replication.databases to an explicit list excluding the database takes the else branch
and fixes it. But unlisted databases then get replicateByDefault: false, so any table relying on
the default rather than an explicit replicate: true would silently stop replicating — including
internal databases if the operator's list is incomplete. That is a large footgun to hand someone
whose only goal is to stop a log flood.
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 83
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HarperFast/harper-pro
-
Difficulty 2/5 Half a day Newbie friendliness 88/100
HarperFast/harper-pro#836 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
HarperFast/harper-pro#728 ·
-
area:replication enhancement feature:plugin-substrate
Difficulty 5/5 Over a week Newbie friendliness 25/100
HarperFast/harper-pro#886 ·
-
area:replication enhancement feature:plugin-substrate
Difficulty 3/5 1-2 days Newbie friendliness 68/100
HarperFast/harper-pro#885 ·
-
area:replication enhancement feature:plugin-substrate
Difficulty 5/5 Over a week Newbie friendliness 25/100
HarperFast/harper-pro#884 ·
All issues in HarperFast/harper-pro
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/skills#96 ·
-
[Block] Latest Posts [Type] Bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
sugarlabs/musicblocks#8847 ·