Store dedup hardlinks empty `.fvs2/lock` files together, causing `cpak update` to hang

Aperta
#90 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, linux
Ambito
backend, databases

Direzione di ricerca

Start by tracing cpak's deduplication of empty files and the code that creates and acquires .fvs2/lock and blocks/.objects locks; the issue does not name source files or tests. Reproduce with hardlinked lock files and check whether dedup preserves distinct lock inodes. Done when cpak update and cpak audit complete without self-deadlock, and lock acquisition reports a clear timeout.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

On my store some of the empty .fvs2/lock files in store/fvs/layers/*/ were merged into a single inode (hardlinks), and the two lock files in store/fvs/blocks/.objects/ also shared an inode. flock locks belong to the inode, so locks that cpak treats as separate, all collided. This caused:

  1. cpak update to hang forever after the last "already present in the store, skipping.." line
  2. cpak audit to fail with repo lock timeout

After giving each lock file its own inode back, both commands worked again and the update completed.

Environment

  • cpak v2.14.18
  • Fedora KDE 45 with Plasma 6.8.2 on x86_64
  • cpak doctor: all checks are okay, the only WARN is "managed host lockdown: enforcement=off"
  • Package I wanted to update: github.com/bottlesdevs/bottles

History

  1. The first cpak update github.com/bottlesdevs/bottles failed with:
    repo is locked by another process (lock: .../store/fvs/layers/2f6774883531.../.fvs2/lock).
  2. Every later run hung after listing all layers as already present. Killing cpak and cpak-storaged and retrying made no difference.
  3. I did not run cpak dedup before starting.

Evidence: update hanging

While hung, cpak and cpak-storaged were at 0% CPU, with no open sockets and almost no I/O.

  • strace -f -p <cpak-storaged>: one thread blocked in flock(10, LOCK_EX); all other threads idle in futex/epoll.
  • lsof showed only cpak-storaged holding the lock files, with fd 9 = fvs/blocks/.objects/transaction.lock and fd 10 = fvs/blocks/.objects/lock.
  • /proc/locks for inode 4870203:
    58: FLOCK ADVISORY READ  19850 00:24:4870203 0 EOF
    58: -> FLOCK ADVISORY WRITE 19850 00:24:4870203 0 EOF
    
    PID 19850 was cpak-storaged. The same process holds a shared lock on this inode and waits for an exclusive lock on it, so the request can never be granted.
  • stat showed that .objects/lock and .objects/transaction.lock have the same inode (4870203).

Evidence: audit failure

cpak audit, with no other cpak process running, failed with:

Error: audit: garbage collection failed: repo is locked by another process
(lock: .../store/fvs/layers/0cfc916b8a9b65764ab8fd70518ab15420015e27b2a0942557782f86a20d6d03/.fvs2/lock): repo lock timeout
  • /proc/locks showed only the audit's own PID holding an exclusive flock on that inode (4875074), with no other holder and no process waiting on it.
  • strace -f -y -tt -e trace=... of the audit shows one thread opening that lock file at 22:50:21.04, retrying flock(fd, LOCK_EX|LOCK_NB) every ~50 ms (each returning EAGAIN, "resource temporarily unavailable"), and closing the file exactly 5 s later at 22:50:26.04. That matches a 5 s lock timeout.
  • No successful flock on a path containing 0cfc916b appears anywhere in the trace, yet that inode was locked by the same process. The lock was taken through a different path to the same inode.

Root cause

stat -c '%h %i %s %n' on all lock files before the workaround:

3  4870203 0 .../store/fvs/blocks/.objects/lock
3  4870203 0 .../store/fvs/blocks/.objects/transaction.lock
16 4875074 0 .../store/fvs/layers/<15 different layers>/.fvs2/lock    (0cfc916b..., 014dc120..., 2930d0c0..., ...)
1  <own inode> 0 .../store/fvs/layers/<9 other layers>/.fvs2/lock      (3e0e3c6f..., f8d3df10..., c2b412a5...,
                                                                         27e4f200..., a5eda581..., ad65a596...,
                                                                         fc66731c..., e12bebc0..., 2f677488...)

The link count of 16 is most likely the 15 layer locks plus one link in the dabadee object store (I confirmed that link for the .objects inode, not for this one).

  • All lock files are empty (size 0).
  • find ~/.local/share/cpak -xdev -inum 4870203 showed the extra link on .objects/lock:
    cpak/dabadee/.dabadee-v2/objects/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-<suffix>.
    That hash is the SHA-256 of an empty file, so the dedup store (dabadee) is replacing identical empty files with hardlinks, including the lock files.
  • The nine layers with link count 1 are the newest ones, including 2f677488... from the first failed update. This is consistent with dedup running over older layers after they are created.

Because flock locks are per inode, "layer A's lock" and "layer B's lock" (or transaction.lock and lock) are the same lock. cpak takes one, then waits for the other, and waits on itself.

Workaround (works for me)

With cpak-storaged and all cpak processes stopped, replace each hardlinked lock file with its own copy:

for f in ~/.local/share/cpak/store/fvs/layers/*/.fvs2/lock; do
  if [ "$(stat -c %h "$f")" -gt 1 ]; then cp -p "$f" "$f.new" && mv "$f.new" "$f"; fi
done
o=~/.local/share/cpak/store/fvs/blocks/.objects
cp -p "$o/transaction.lock" "$o/transaction.lock.new" && mv "$o/transaction.lock.new" "$o/transaction.lock"

Afterwards every layer lock and transaction.lock has link count 1 (.objects/lock still has link count 2 because of the dabadee object). cpak audit finishes with no lock error, and cpak update github.com/bottlesdevs/bottles completes (Checked 1 cpak(s): 1 updated). A second cpak audit was also clean.

After the update and a clean cpak audit, all 24 layer locks and transaction.lock have link count 1 and a unique inode, only .objects/lock has link count 2 (the dabadee link). Nothing was re-merged by the update itself.

To check for the problem on any store:

stat -c '%h %i %n' ~/.local/share/cpak/store/fvs/layers/*/.fvs2/lock \
  ~/.local/share/cpak/store/fvs/blocks/.objects/*lock | sort -k2 -n

Repeated inode numbers or link counts above 1 on these files mean the problem is present.

Expected behaviour

  • Dedup should skip lock files (or all empty files in .fvs2/ and blocks/.objects/), or locking should not depend on inode identity.
  • A lock that cannot be acquired should produce a clear error after a timeout instead of an indefinite hang in cpak-storaged.

Notes

I used Claude while debugging this, this issue has been proofread by me and I take full responsibility.

Lingua principale
Go
Stelle
218
Fork
5
Merge medio
45m
PR unite (30g)
5

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Containerpak/cpak

Tutte le issue di Containerpak/cpak

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.