release: run smoke checks in a workflow on a self-hosted runner

Open
  • Is something wrong?
#388 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
github-actions, go
Domain
ci-cd, distributed-systems, release

Research direction

Start with the release workflow configuration and doc/release-guide.md, then inspect the Makefile targets for the listed checks and the sweep command for the cluster run. Implement the workflow for workflow_dispatch and release/v* pushes, with commit and runner access safeguards, pass criteria, and uploaded reports. Done means the workflow gates publishing as documented and its checks and cluster smoke run produce the required artifacts and results.

Written by the indexing model from the issue text.

Description

Goal

Add release smoke checks to the release process.
Run them in a GitHub workflow on a self-hosted runner that has access to our cluster.

Why

gorums-release prepare runs make test and make testrace, and CI runs make test and golangci-lint.
Nothing runs these checks before a release:

  • make integrationtest (real TCP; the default tests use in-memory bufconn)
  • make stresstest (ordering under load)
  • make lint, make modernize, make goplscheck
  • tests of the benchkit and examples modules (check that make test covers them)
  • a multi-host run on the real cluster, which finds connection, reconnect, and clock-sync problems

Proposal

  1. Add a workflow, triggered by workflow_dispatch and by a push to release/v* branches.
    It runs on a self-hosted runner on the cluster.
  2. The workflow runs the local checks above, then the cluster smoke run:
    sweep -test 9 -sweep stream-dedup-smoke -stream-mode dual,dedup -benchmarks SymmetricQuorumCall,SymmetricMulticast.
  3. The workflow checks the commit it builds: it must be the release candidate commit, not a feature branch.
  4. Pass criteria:
    • every manifest has "status": "succeeded";
    • no degraded reps;
    • agg.csv has non-zero throughput and a latency for every row;
    • clock-offset drift in sweep.log is small.
      The empty SymmetricMulticast latency time series is a known issue and is not a failure until it is fixed.
  5. Upload the sweep report (report.pdf, agg.csv, logs) as workflow artifacts.
  6. Update doc/release-guide.md: add a "Smoke Checks" step between "Wait for CI" and "Publish", and require the workflow to pass before gorums-release publish.

Open questions

  • How does the runner reach the cluster, and who maintains it?
  • Should publish refuse to run unless the smoke workflow passed on the release commit?
  • Self-hosted runners on a public repository: restrict the workflow to release/* branches and trusted actors, so that a fork PR cannot run code on the cluster.
Dominant language
Go
Stars
149
Forks
15
Avg merge
1d 2h
Merged PRs (30d)
50

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from relab/gorums

All issues in relab/gorums

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.