Store dedup hardlinks empty `.fvs2/lock` files together, causing `cpak update` to hang
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
Research direction
Start by tracing cpak's deduplication of empty files and the code that creates and acquires .fvs2/lock and blocks/.objects locks; the issue does not name source files or tests. Reproduce with hardlinked lock files and check whether dedup preserves distinct lock inodes. Done when cpak update and cpak audit complete without self-deadlock, and lock acquisition reports a clear timeout.
Written by the indexing model from the issue text.
Description
Summary
On my store some of the empty .fvs2/lock files in store/fvs/layers/*/ were merged into a single inode (hardlinks), and the two lock files in store/fvs/blocks/.objects/ also shared an inode. flock locks belong to the inode, so locks that cpak treats as separate, all collided. This caused:
cpak updateto hang forever after the last "already present in the store, skipping.." linecpak auditto fail withrepo lock timeout
After giving each lock file its own inode back, both commands worked again and the update completed.
Environment
- cpak v2.14.18
- Fedora KDE 45 with Plasma 6.8.2 on x86_64
cpak doctor: all checks are okay, the only WARN is "managed host lockdown: enforcement=off"- Package I wanted to update: github.com/bottlesdevs/bottles
History
- The first
cpak update github.com/bottlesdevs/bottlesfailed with:
repo is locked by another process (lock: .../store/fvs/layers/2f6774883531.../.fvs2/lock). - Every later run hung after listing all layers as already present. Killing
cpakandcpak-storagedand retrying made no difference. - I did not run
cpak dedupbefore starting.
Evidence: update hanging
While hung, cpak and cpak-storaged were at 0% CPU, with no open sockets and almost no I/O.
strace -f -p <cpak-storaged>: one thread blocked inflock(10, LOCK_EX); all other threads idle infutex/epoll.lsofshowed onlycpak-storagedholding the lock files, with fd 9 =fvs/blocks/.objects/transaction.lockand fd 10 =fvs/blocks/.objects/lock./proc/locksfor inode 4870203:
PID 19850 was58: FLOCK ADVISORY READ 19850 00:24:4870203 0 EOF 58: -> FLOCK ADVISORY WRITE 19850 00:24:4870203 0 EOFcpak-storaged. The same process holds a shared lock on this inode and waits for an exclusive lock on it, so the request can never be granted.statshowed that.objects/lockand.objects/transaction.lockhave the same inode (4870203).
Evidence: audit failure
cpak audit, with no other cpak process running, failed with:
Error: audit: garbage collection failed: repo is locked by another process
(lock: .../store/fvs/layers/0cfc916b8a9b65764ab8fd70518ab15420015e27b2a0942557782f86a20d6d03/.fvs2/lock): repo lock timeout
/proc/locksshowed only the audit's own PID holding an exclusive flock on that inode (4875074), with no other holder and no process waiting on it.strace -f -y -tt -e trace=...of the audit shows one thread opening that lock file at 22:50:21.04, retryingflock(fd, LOCK_EX|LOCK_NB)every ~50 ms (each returningEAGAIN, "resource temporarily unavailable"), and closing the file exactly 5 s later at 22:50:26.04. That matches a 5 s lock timeout.- No successful
flockon a path containing0cfc916bappears anywhere in the trace, yet that inode was locked by the same process. The lock was taken through a different path to the same inode.
Root cause
stat -c '%h %i %s %n' on all lock files before the workaround:
3 4870203 0 .../store/fvs/blocks/.objects/lock
3 4870203 0 .../store/fvs/blocks/.objects/transaction.lock
16 4875074 0 .../store/fvs/layers/<15 different layers>/.fvs2/lock (0cfc916b..., 014dc120..., 2930d0c0..., ...)
1 <own inode> 0 .../store/fvs/layers/<9 other layers>/.fvs2/lock (3e0e3c6f..., f8d3df10..., c2b412a5...,
27e4f200..., a5eda581..., ad65a596...,
fc66731c..., e12bebc0..., 2f677488...)
The link count of 16 is most likely the 15 layer locks plus one link in the dabadee object store (I confirmed that link for the .objects inode, not for this one).
- All lock files are empty (size 0).
find ~/.local/share/cpak -xdev -inum 4870203showed the extra link on.objects/lock:
cpak/dabadee/.dabadee-v2/objects/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-<suffix>.
That hash is the SHA-256 of an empty file, so the dedup store (dabadee) is replacing identical empty files with hardlinks, including the lock files.- The nine layers with link count 1 are the newest ones, including
2f677488...from the first failed update. This is consistent with dedup running over older layers after they are created.
Because flock locks are per inode, "layer A's lock" and "layer B's lock" (or transaction.lock and lock) are the same lock. cpak takes one, then waits for the other, and waits on itself.
Workaround (works for me)
With cpak-storaged and all cpak processes stopped, replace each hardlinked lock file with its own copy:
for f in ~/.local/share/cpak/store/fvs/layers/*/.fvs2/lock; do
if [ "$(stat -c %h "$f")" -gt 1 ]; then cp -p "$f" "$f.new" && mv "$f.new" "$f"; fi
done
o=~/.local/share/cpak/store/fvs/blocks/.objects
cp -p "$o/transaction.lock" "$o/transaction.lock.new" && mv "$o/transaction.lock.new" "$o/transaction.lock"
Afterwards every layer lock and transaction.lock has link count 1 (.objects/lock still has link count 2 because of the dabadee object). cpak audit finishes with no lock error, and cpak update github.com/bottlesdevs/bottles completes (Checked 1 cpak(s): 1 updated). A second cpak audit was also clean.
After the update and a clean cpak audit, all 24 layer locks and transaction.lock have link count 1 and a unique inode, only .objects/lock has link count 2 (the dabadee link). Nothing was re-merged by the update itself.
To check for the problem on any store:
stat -c '%h %i %n' ~/.local/share/cpak/store/fvs/layers/*/.fvs2/lock \
~/.local/share/cpak/store/fvs/blocks/.objects/*lock | sort -k2 -n
Repeated inode numbers or link counts above 1 on these files mean the problem is present.
Expected behaviour
- Dedup should skip lock files (or all empty files in
.fvs2/andblocks/.objects/), or locking should not depend on inode identity. - A lock that cannot be acquired should produce a clear error after a timeout instead of an indefinite hang in
cpak-storaged.
Notes
I used Claude while debugging this, this issue has been proofread by me and I take full responsibility.
- Dominant language
- Go
- Stars
- 218
- Forks
- 5
- Avg merge
- 45m
- Merged PRs (30d)
- 5
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Containerpak/cpak
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Containerpak/cpak#93 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Containerpak/cpak#92 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
Containerpak/cpak#89 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 58/100
Containerpak/cpak#88 ·
-
[Feature] Ability to install applications sytem wide (Like how you can do it with `flatpak --system`Open
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Containerpak/cpak#81 ·
All issues in Containerpak/cpak
Similar issues
-
Type: Bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
projectdiscovery/httpx#2735 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
charmbracelet/huh#849 ·
-
help wanted stage/010-actionable
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
hivecommons/hive#18458 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
suzuki-shunsuke/tfcmt#2636 ·
Maintainers usually reply within 1 day
-
ui: links opened in a new tab omit the public URL path prefix (e.g. rotations on a schedule's assignments page)Possibly taken @dginther claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day