[Bug]: CouchDB 3.5.1 upgrade leads to elevated memory and instability (OOM on single node + full cluster drop during rolling upgrade)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- aws, erlang
- Domain
- cloud, databases, distributed-systems
Research direction
No source file, test, or entry point is named. Start by reproducing the 3.4.2-to-3.5.1 upgrade with Erlang/OTP 26 under high shard and database counts, then measure beam.smp memory, process counts, compaction, and rolling node cycling; done means identifying the regression and preventing OOM or cluster-wide unavailability.
Written by the indexing model from the issue text.
Description
Version
3.5.1
Describe the problem you're encountering
After upgrading from CouchDB 3.4.2 -> 3.5.1, we are seeing stability regressions in both:
- Single node environments (OOM kill crash)
- 3 node clustered environments (full cluster outage during rolling upgrade)
These environments were stable on 3.4.2 under same workloads
The main symptoms include: - Increased Erlang VM RSS memory usage (beam.smp)
- Very large Erlang process counts post-upgrade
- CPU spikes during compaction/rebalancing
- Cluster becoming unavailable during node cycling
Expected Behaviour
- CouchDB 3.5.1 should not significantly increase baseline memory/process usage on the same workload
- Single node instances should not be OOM killed under normal compaction load
- Rolling node upgrades should not result in full cluster outage
Steps to Reproduce
We have observed the issues under the following conditions:
- Upgrade CouchDB 3.4.2 → 3.5.1 (Erlang/OTP 26)
- Run with medium memory nodes (~8GB RAM)
- Large shard count / very high DB count
- Compaction or shard movement occurring
- Cycle one node during rolling upgrade (ASG refresh)
Result: memory/process growth and potential node loss -> cluster unavailability
Your Environment
Deployment
- AWS EC2 instances with persistent couch_data
- CouchDB nodes managed via ASG refresh (one node at a time)
- HAProxy + Fauxton access through fronted endpoint
Cluster configuration:
- 3-node cluster (n=3, q=1)
- Placement: primary / secondary / trinary
- Data size: ~319k databases
- Disk usage: ~640GB used of ~1TB per node
Instance resources:
~7.6GB RAM
Additional Context
No response
- Dominant language
- Erlang
- Stars
- 7k
- Forks
- 1.1k
- Avg merge
- 2d 24m
- Merged PRs (30d)
- 12
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/couchdb
-
bug needs-triage
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 40/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 45/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 45/100
-
enhancement needs-triage
Difficulty 3/5 1-2 days Newbie friendliness 55/100
Similar issues
-
samples
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
area-deployment area-integrations triage:bot-seen
Difficulty 2/5 Half a day Newbie friendliness 86/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100