artifact: share in-flight file fetches in the cache, and make the local loader's read cancellable

Open
#387 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go

Research direction

Start by locating cachingLoader.Load, cachingLoader.warm, and local.FileLoader.Load in the artifact package, then read their existing tests and the S3 and GitHub loaders' cancellation behavior. Add coverage for concurrent loads of the same path and for a local read whose caller cancels; done when warm-up and lookup share one fetch, and a cancelled local load returns promptly.

Written by the indexing model from the issue text.

Description

Type/Improvement
Current Limitation

Two gaps in artifact that show up when a host warms the file cache in the background with Set.Warm (OpenNSW/nsw-srilanka#1121):

  1. No shared in-flight fetches. cachingLoader.Load and cachingLoader.warm each fetch from the next loader when a path is missing from the cache. A lookup that arrives while the warm-up is fetching the same file fetches it a second time. The lookup is no slower than without warming, but it gains nothing from the warm-up, and the source sees duplicate requests.
  2. The local loader can't be cancelled. local.FileLoader.Load ignores its context and calls os.ReadFile. A read that stalls (for example, on a hung network mount) can't be interrupted, so a caller that cancels Warm and waits for it can wait indefinitely. The S3 and GitHub loaders already honour cancellation.
Suggested Improvement
  1. Share in-flight fetches per path in cachingLoader, for example with golang.org/x/sync/singleflight, so Load and warm wait on one fetch. A caller whose context ends should stop waiting without cancelling the fetch for the others.
  2. Let local.FileLoader.Load return when its context ends, even if the read underneath it carries on. A context check before os.ReadFile isn't enough once the read has started.
Version

v0.0.0-20261104115811-4805eeda5c66

Additional Context

nsw-srilanka bounds its shutdown wait for the warm-up at five seconds, so (2) can't hang its shutdown. Both are improvements, not blockers.

Dominant language
Go
Stars
6
Forks
8
Avg merge
21h 36m
Merged PRs (30d)
42

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenNSW/core

All issues in OpenNSW/core

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.