JuliaImages/Images.jl

Add a thorough test on **all** compatible versions

Open

#857 创建于 2020年1月1日

在 GitHub 查看
 (1 评论) (0 反应) (0 负责人)Julia (142 fork)batch import
help wanted

仓库指标

Star
 (541 star)
PR 合并指标
 (30 天内没有已合并 PR)

描述

In #805 I once proposed to test the nightly stage on the master branch of all dependencies, it could help catch some errors before releasing a new version without requiring new CI resources. It turned out that the proposal isn't really useful in practice if we choose to tag our versions frequently.

Recent devs showed that many bugs are due to incompatible versions that we claim compatible. E.g., https://github.com/JuliaImages/ImageCore.jl/pull/110, https://github.com/JuliaGraphics/Colors.jl/pull/390, https://github.com/JuliaGraphics/Colors.jl/pull/383

That's not good news for both users and developers. Code reviews can help catch some of them but to guarantee none of them passes silently, we might need a really thorough test on it.

The plan is simple:

  • generating all possible Manifest.tomls for all versions of dependencies we claim compatible
  • test all of them using CI

The only and really big challenge is we don't have enough CI resources to support this.

There could be thousands of possible combinations if we literally test all of them and it's impractical, hence we need to take some strategy to reduce the number, for example:

  • versions that older than three months are not under consideration
  • if there're many patch versions, only test the latest, e.g., test v0.8.7 and ignore v0.8.0-v0.8.6.
  • only trigger the CI when we trigger the registrator

I need to do some statistics on this strategy and see if it reduces the number to a practical one.

Right now, this isn't an urgent issue, and it might not be the best time to tackle this.

贡献者指南