FluxML/Flux.jl

Separate nightly CI from posted CI badge

Open

#2237 opened on Apr 23, 2023

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Julia (4,725 stars) (619 forks)batch import
discussiongood first issuetesting

Description

Inspired from a discussion on Slack about how other packages handle the same. Flux frequently has to deal with this because new nightly releases often break Zygote, but Flux itself is rarely affected. There seem to be two main approaches to accomplishing this:

  1. Allow the nightly CI job to fail, e.g. https://github.com/JuliaSmoothOptimizers/SolverTest.jl/pull/6. This is simple to implement but does risk missing real failures if one doesn't dig beyond the big green checkmark.
  2. Separate nightly CI into its own job and show the non-nightly results badge, e.g. https://github.com/JuliaManifolds/Manifolds.jl/blob/master/.github/workflows/nightly.yml. This is more work but will still fail loudly for PRs.

A related question is whether we want to do the same for downstream tests. There are some known downstreams which are quite flaky, but since we don't include any downstream results in the README I'd assume this is less of a problem.

Contributor guide

Tech stack
None
Domain
ci cd
Issue type
devops
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Familiarity with GitHub ActionsUnderstanding of CI workflows
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Investigate the current CI workflow in .github/workflows/ci.yml. Review the two approaches mentioned: allowing nightly CI to fail or creating separate nightly workflow. Look at the SolverTest.jl and Manifolds.jl examples for reference. Decide which approach to implement and modify the CI configuration accordingly.