Persist the SNARK test caches across CI runs

Open
  • Is something wrong?
#3,467 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
github-actions, rust
Domain
build-system, ci-cd, cryptography, testing

Research direction

First check the dependency on #3433, then read the test job in .github/workflows/test-rust.yml and the cache readers in circuits/key_provider.rs and circuits/trusted_setup.rs. Compare the duplicate cache paths in proof_system/ivc_halo2_snark/prover_setup.rs and circuits/halo2_ivc/tests/common/generators/setup.rs. Done means warm CI restores safely, stale entries self-heal, only full slow-suite runs save pruned caches, and the issue reports the measured gain.

Written by the indexing model from the issue text.

Description

cryptography :closed_lock_with_key: testing :repeat: to-groom :thinking:

Why

Neither the keygen cache nor the SRS is persisted across CI runs (the toolchain-and-cache action only saves dependency artifacts through Swatinem/rust-cache), so every nightly and every slow PR leg starts cold.

What

Add an actions/cache layer for target/mithril-circuit-test-cache with a three-layer invalidation design: content-keyed directories for semantic staleness, a versions-scoped cache key for serialization format drift, and self-healing readers as backstop.

How

Paths are relative to mithril-stm/src/ unless they start with .github/.

  • Add an actions/cache/restore step for target/mithril-circuit-test-cache in the test job of .github/workflows/test-rust.yml (next to the toolchain-and-cache composite step), with an exact key folding a manual generation token and a hash of the midnight-* entries of Cargo.lock
  • Scope every restore-keys prefix to that versions hash so a midnight-proofs upgrade (serialization format drift) can never restore stale bytes
  • Save with actions/cache/save only from the runs that execute the full slow suite (the include-slow-tests: true invocation from nightly-dispatcher.yml); PR runs stay restore-only
  • Make the readers self-healing: on deserialization failure, KeyProvider::cache_state and read_optional (circuits/key_provider.rs) and TrustedSetupProvider::get_trusted_setup_parameters (circuits/trusted_setup.rs) should delete the offending file and regenerate instead of failing the test
  • Prune fingerprint directories not touched by the current run before saving, so retired fingerprints do not accumulate in the archive
  • Make one recursive verifying-key cache canonical for both the circuit tests and IvcSnarkProverSetup::build_for_test_degree, with the prover path storing only the additional proving key: today proof_system/ivc_halo2_snark/prover_setup.rs and circuits/halo2_ivc/tests/common/generators/setup.rs hold the same key under two protocols, because KeyProvider::verification_key treats a missing proving key as a miss and regenerates the pair
  • Assess the gain (expected: removes the cold start from warm CI runs) and report it in the issue

Depends on #3433.

Dominant language
Rust
Stars
155
Forks
52
Avg merge
2d 17h
Merged PRs (30d)
37

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from IntersectMBO/mithril

All issues in IntersectMBO/mithril

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.