artifact: share in-flight file fetches in the cache, and make the local loader's read cancellable
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go
- Domain
- backend, performance
Research direction
Start by locating cachingLoader.Load, cachingLoader.warm, and local.FileLoader.Load in the artifact package, then read their existing tests and the S3 and GitHub loaders' cancellation behavior. Add coverage for concurrent loads of the same path and for a local read whose caller cancels; done when warm-up and lookup share one fetch, and a cancelled local load returns promptly.
Written by the indexing model from the issue text.
Description
Current Limitation
Two gaps in artifact that show up when a host warms the file cache in the background with Set.Warm (OpenNSW/nsw-srilanka#1121):
- No shared in-flight fetches.
cachingLoader.LoadandcachingLoader.warmeach fetch from the next loader when a path is missing from the cache. A lookup that arrives while the warm-up is fetching the same file fetches it a second time. The lookup is no slower than without warming, but it gains nothing from the warm-up, and the source sees duplicate requests. - The local loader can't be cancelled.
local.FileLoader.Loadignores its context and callsos.ReadFile. A read that stalls (for example, on a hung network mount) can't be interrupted, so a caller that cancelsWarmand waits for it can wait indefinitely. The S3 and GitHub loaders already honour cancellation.
Suggested Improvement
- Share in-flight fetches per path in
cachingLoader, for example withgolang.org/x/sync/singleflight, soLoadandwarmwait on one fetch. A caller whose context ends should stop waiting without cancelling the fetch for the others. - Let
local.FileLoader.Loadreturn when its context ends, even if the read underneath it carries on. A context check beforeos.ReadFileisn't enough once the read has started.
Version
v0.0.0-20261104115811-4805eeda5c66
Additional Context
nsw-srilanka bounds its shutdown wait for the warm-up at five seconds, so (2) can't hang its shutdown. Both are improvements, not blockers.
- Dominant language
- Go
- Stars
- 6
- Forks
- 8
- Avg merge
- 21h 36m
- Merged PRs (30d)
- 42
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenNSW/core
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
payment: test fixtures hardcode real gateway names (govpay/lankapay) instead of generic placeholdersOpen
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
nsw-sl cleanup Type/Improvement
Difficulty 3/5 1-2 days Newbie friendliness 48/100
Maintainers usually reply within 1 day
-
remote: let the host inject a logger, and log with one consistent policyPossibly taken @mushrafmim claimed this 5 days ago. OpenType/Improvement
Difficulty 4/5 3-5 days Newbie friendliness 22/100
Maintainers usually reply within 1 day
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day