Store dedup hardlinks empty `.fvs2/lock` files together, causing `cpak update` to hang
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 48/100
Direzione di ricerca
Start by tracing cpak's deduplication of empty files and the code that creates and acquires .fvs2/lock and blocks/.objects locks; the issue does not name source files or tests. Reproduce with hardlinked lock files and check whether dedup preserves distinct lock inodes. Done when cpak update and cpak audit complete without self-deadlock, and lock acquisition reports a clear timeout.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
On my store some of the empty .fvs2/lock files in store/fvs/layers/*/ were merged into a single inode (hardlinks), and the two lock files in store/fvs/blocks/.objects/ also shared an inode. flock locks belong to the inode, so locks that cpak treats as separate, all collided. This caused:
cpak updateto hang forever after the last "already present in the store, skipping.." linecpak auditto fail withrepo lock timeout
After giving each lock file its own inode back, both commands worked again and the update completed.
Environment
- cpak v2.14.18
- Fedora KDE 45 with Plasma 6.8.2 on x86_64
cpak doctor: all checks are okay, the only WARN is "managed host lockdown: enforcement=off"- Package I wanted to update: github.com/bottlesdevs/bottles
History
- The first
cpak update github.com/bottlesdevs/bottlesfailed with:
repo is locked by another process (lock: .../store/fvs/layers/2f6774883531.../.fvs2/lock). - Every later run hung after listing all layers as already present. Killing
cpakandcpak-storagedand retrying made no difference. - I did not run
cpak dedupbefore starting.
Evidence: update hanging
While hung, cpak and cpak-storaged were at 0% CPU, with no open sockets and almost no I/O.
strace -f -p <cpak-storaged>: one thread blocked inflock(10, LOCK_EX); all other threads idle infutex/epoll.lsofshowed onlycpak-storagedholding the lock files, with fd 9 =fvs/blocks/.objects/transaction.lockand fd 10 =fvs/blocks/.objects/lock./proc/locksfor inode 4870203:
PID 19850 was58: FLOCK ADVISORY READ 19850 00:24:4870203 0 EOF 58: -> FLOCK ADVISORY WRITE 19850 00:24:4870203 0 EOFcpak-storaged. The same process holds a shared lock on this inode and waits for an exclusive lock on it, so the request can never be granted.statshowed that.objects/lockand.objects/transaction.lockhave the same inode (4870203).
Evidence: audit failure
cpak audit, with no other cpak process running, failed with:
Error: audit: garbage collection failed: repo is locked by another process
(lock: .../store/fvs/layers/0cfc916b8a9b65764ab8fd70518ab15420015e27b2a0942557782f86a20d6d03/.fvs2/lock): repo lock timeout
/proc/locksshowed only the audit's own PID holding an exclusive flock on that inode (4875074), with no other holder and no process waiting on it.strace -f -y -tt -e trace=...of the audit shows one thread opening that lock file at 22:50:21.04, retryingflock(fd, LOCK_EX|LOCK_NB)every ~50 ms (each returningEAGAIN, "resource temporarily unavailable"), and closing the file exactly 5 s later at 22:50:26.04. That matches a 5 s lock timeout.- No successful
flockon a path containing0cfc916bappears anywhere in the trace, yet that inode was locked by the same process. The lock was taken through a different path to the same inode.
Root cause
stat -c '%h %i %s %n' on all lock files before the workaround:
3 4870203 0 .../store/fvs/blocks/.objects/lock
3 4870203 0 .../store/fvs/blocks/.objects/transaction.lock
16 4875074 0 .../store/fvs/layers/<15 different layers>/.fvs2/lock (0cfc916b..., 014dc120..., 2930d0c0..., ...)
1 <own inode> 0 .../store/fvs/layers/<9 other layers>/.fvs2/lock (3e0e3c6f..., f8d3df10..., c2b412a5...,
27e4f200..., a5eda581..., ad65a596...,
fc66731c..., e12bebc0..., 2f677488...)
The link count of 16 is most likely the 15 layer locks plus one link in the dabadee object store (I confirmed that link for the .objects inode, not for this one).
- All lock files are empty (size 0).
find ~/.local/share/cpak -xdev -inum 4870203showed the extra link on.objects/lock:
cpak/dabadee/.dabadee-v2/objects/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-<suffix>.
That hash is the SHA-256 of an empty file, so the dedup store (dabadee) is replacing identical empty files with hardlinks, including the lock files.- The nine layers with link count 1 are the newest ones, including
2f677488...from the first failed update. This is consistent with dedup running over older layers after they are created.
Because flock locks are per inode, "layer A's lock" and "layer B's lock" (or transaction.lock and lock) are the same lock. cpak takes one, then waits for the other, and waits on itself.
Workaround (works for me)
With cpak-storaged and all cpak processes stopped, replace each hardlinked lock file with its own copy:
for f in ~/.local/share/cpak/store/fvs/layers/*/.fvs2/lock; do
if [ "$(stat -c %h "$f")" -gt 1 ]; then cp -p "$f" "$f.new" && mv "$f.new" "$f"; fi
done
o=~/.local/share/cpak/store/fvs/blocks/.objects
cp -p "$o/transaction.lock" "$o/transaction.lock.new" && mv "$o/transaction.lock.new" "$o/transaction.lock"
Afterwards every layer lock and transaction.lock has link count 1 (.objects/lock still has link count 2 because of the dabadee object). cpak audit finishes with no lock error, and cpak update github.com/bottlesdevs/bottles completes (Checked 1 cpak(s): 1 updated). A second cpak audit was also clean.
After the update and a clean cpak audit, all 24 layer locks and transaction.lock have link count 1 and a unique inode, only .objects/lock has link count 2 (the dabadee link). Nothing was re-merged by the update itself.
To check for the problem on any store:
stat -c '%h %i %n' ~/.local/share/cpak/store/fvs/layers/*/.fvs2/lock \
~/.local/share/cpak/store/fvs/blocks/.objects/*lock | sort -k2 -n
Repeated inode numbers or link counts above 1 on these files mean the problem is present.
Expected behaviour
- Dedup should skip lock files (or all empty files in
.fvs2/andblocks/.objects/), or locking should not depend on inode identity. - A lock that cannot be acquired should produce a clear error after a timeout instead of an indefinite hang in
cpak-storaged.
Notes
I used Claude while debugging this, this issue has been proofread by me and I take full responsibility.
- Lingua principale
- Go
- Stelle
- 218
- Fork
- 5
- Merge medio
- 45m
- PR unite (30g)
- 5
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di Containerpak/cpak
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
Containerpak/cpak#93 ·
-
fish completions are brokenAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
Containerpak/cpak#92 ·
-
displayX11 requires Xephyr on X11 but nested display fails with MIT-SHM errors (black screen)Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 52/100
Containerpak/cpak#89 ·
-
Difficoltà 3/5 1-2 giorni Idoneità per principianti 58/100
Containerpak/cpak#88 ·
-
[Feature] Ability to install applications sytem wide (Like how you can do it with `flatpak --system`Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
Containerpak/cpak#81 ·
Tutte le issue di Containerpak/cpak
Issue simili
-
bug triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 1 giorno
-
bug laptop
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 82/100
router-for-me/CLIProxyAPI#7607 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement ui/ux
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
myNameArnav/lex#52 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100