discussiongood first issuetesting
描述
Inspired from a discussion on Slack about how other packages handle the same. Flux frequently has to deal with this because new nightly releases often break Zygote, but Flux itself is rarely affected. There seem to be two main approaches to accomplishing this:
- Allow the nightly CI job to fail, e.g. https://github.com/JuliaSmoothOptimizers/SolverTest.jl/pull/6. This is simple to implement but does risk missing real failures if one doesn't dig beyond the big green checkmark.
- Separate nightly CI into its own job and show the non-nightly results badge, e.g. https://github.com/JuliaManifolds/Manifolds.jl/blob/master/.github/workflows/nightly.yml. This is more work but will still fail loudly for PRs.
A related question is whether we want to do the same for downstream tests. There are some known downstreams which are quite flaky, but since we don't include any downstream results in the README I'd assume this is less of a problem.