[Bug]: Replication result depends on whether the target has a validate_doc_update function
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 44/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- erlang, javascript
- Ambito
- database, databases, distributed-systems
Direzione di ricerca
Start with couch_db:before_docs_update and prep_and_validate_replicated_updates, then trace how couch_replicator_worker:find_missing handles the missing ratio. Run the provided curl and jq reproduction against CouchDB 3.5.3 and compare the final document in t1 and t2. Done means both targets reach the same document state with and without the accepting validate_doc_update design doc.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Version
3.5.3
Describe the problem you're encountering
Two target databases that get the same replications, in the same order, from the same sources can end up with a different state for a document: one ends deleted, the other ends live with an old revision as the winner. The only difference between the targets is a design doc whose validate_doc_update accepts everything.
It shows up when a target first receives a revision with a short history (from a replica with a low revs_limit) and later receives the same revision again with its full history.
Expected Behaviour
I would expect replication to give the same result whether or not the target has a validate_doc_update function that accepts the document.
Steps to Reproduce
Needs curl and jq. Single node, one-shot _replicate, run in order.
COUCH=http://adm:[email protected]:5984
J='Content-Type: application/json'
for db in a c b d t1 t2; do curl -sS -X PUT "$COUCH/$db" > /dev/null; done
curl -sS -X PUT "$COUCH/b/_revs_limit" -d 1 > /dev/null
curl -sS -X PUT "$COUCH/d/_revs_limit" -d 2 > /dev/null
curl -sS -X PUT "$COUCH/t2/_design/accept" -H "$J" \
-d '{"validate_doc_update":"function(newDoc, oldDoc, userCtx, secObj) {}"}' > /dev/null
rep() { curl -sS -X POST "$COUCH/_replicate" -H "$J" \
-d "{\"source\":\"$COUCH/$1\",\"target\":\"$COUCH/$2\"${3:+,$3}}" > /dev/null; }
ONLY_X='"doc_ids":["X"]'
# create X (1-a) on a, replicate to c
REV=$(curl -sS -X PUT "$COUCH/a/X" -H "$J" -d '{"v":"a"}' | jq -r .rev)
rep a c "$ONLY_X"
# 6 unrelated docs on a
curl -sS -X POST "$COUCH/a/_bulk_docs" -H "$J" -d '{"docs":[
{"_id":"f1"},{"_id":"f2"},{"_id":"f3"},{"_id":"f4"},{"_id":"f5"},{"_id":"f6"}]}' > /dev/null
# edit X (2-b) on a, replicate to b (revs_limit 1, so b keeps 2-b with no ancestors)
REV=$(curl -sS -X PUT "$COUCH/a/X" -H "$J" -d "{\"_rev\":\"$REV\",\"v\":\"b\"}" | jq -r .rev)
rep a b "$ONLY_X"
# both targets: c then b, giving two separate roots 1-a and 2-b
for t in t1 t2; do rep c $t; rep b $t; done
# both targets: replicate a with one worker, batch size 1
for t in t1 t2; do rep a $t '"worker_processes":1,"worker_batch_size":1'; done
# delete X (3-d) on a, replicate to d (revs_limit 2), then d to both targets
curl -sS -X DELETE "$COUCH/a/X?rev=$REV" > /dev/null
rep a d "$ONLY_X"
for t in t1 t2; do rep d $t; done
curl -sS "$COUCH/t1/X"
curl -sS "$COUCH/t2/X"
Output on 3.5.3:
{"error":"not_found","reason":"deleted"}
{"_id":"X","_rev":"1-cd6dcd82d939d5b6220025fbc573b36a","v":"a"}
Your Environment
CouchDB 3.5.3 (git 7a1f97558), Erlang/OTP 28, single node, q=1 n=1, macOS arm64.
Additional Context
From reading the source, I think two things interact:
couch_db:before_docs_updateonly runsprep_and_validate_replicated_updateswhen the db has a VDU (or a design doc or attachments in the request). That step drops a replicated revision that is already a leaf; without a VDU the same path gets merged.couch_replicator_worker:find_missingskips_revs_diffonce the missing ratio average goes over 0.95 (6 batches in a row where everything was missing), and then re-sends revisions the target already has, with the longer history.
The same split happens with default replication settings when 20,000 unrelated docs come before X in a's changes. If the unrelated docs come after X, or there are only 5, both targets end the same. The relevant files look unchanged on main.
Fuller repro with all runs and controls: https://github.com/blackwell-systems/normalization-confluence/tree/main/research/couchdb-replicator
- Lingua principale
- Erlang
- Stelle
- 7k
- Fork
- 1.1k
- Merge medio
- 10h 16m
- PR unite (30g)
- 35
Preparare l'ambiente
Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di apache/couchdb
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 45/100
apache/couchdb#6166 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Fix: Warning: 'catch ...' is deprecated; please use 'try ... catch ... end' insteadForse di nuovo libera @devx-arjun l’ha presa 10 giorni fa e non c’è nessuna pull request aperta. Apertabeginner-friendly build chore patches-welcome
Difficoltà 4/5 3-5 giorni Idoneità per principianti 45/100
apache/couchdb#6144 · 4 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
bug needs-triage
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
apache/couchdb#6132 · 3 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
[ENHANCEMENT]: Add seq checkpoints to continuous feedsForse già presa @nickva l’ha presa 2 giorni fa. Apertaenhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 45/100
apache/couchdb#6120 · 7 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement needs-triage
Difficoltà 3/5 1-2 giorni Idoneità per principianti 55/100
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di apache/couchdb
Issue simili
-
Filters: ?field:ne=null matches NULL rows instead of non-NULL onesForse già presa @rshdhere l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
Netflix/metaflow-service#504 ·
-
bug domain: database priority: p1 Stellar Wave
Difficoltà 2/5 1-3 ore Idoneità per principianti 74/100
StellarState/SS-backend#632 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
[1.x Bug] run-operation --sql reports success but rolls back writes on transaction-capable adaptersApertatriage
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
dbt-labs/dbt#16784 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Add tests for player queries in backend/src/db/queries.tsForse già presa @Ayinkx l’ha presa oggi. Apertabackend good first issue Stellar Wave
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
OlaGreat/starbound-realms#18 · 1 commento ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
netshot-net/Netshot#413 ·