[BUG] resolve failing and flaky tests in CI, and race conditions in tests using server

未关闭

#1,586 创建于 2025年12月31日 4 条评论 0 个 reaction 已指派 0 人在 GitHub 查看

还没有人认领这个 Issue。

评估

难度: 5/5
预计耗时: 一周以上
新手友好度: 25/100

Issue 类型: 缺陷
描述清晰度: 需要澄清
活跃度: 冷清
技术栈: python
领域: backend, ci-cd, testing

调研方向

Start by reproducing the reported flaky failures in remote CI and inspect the server tests for shared-state or parallel-run races. Review the extension pattern from issue 1575 and compare the proposed mock, local-server, and decoupled server-test approaches. Done means the CI test matrix is reliable and server tests no longer fail sporadically when runs overlap.

由索引模型根据 Issue 内容生成。

描述

bug testing

Currently, the remote CI is not reliable and will result in faliures and sporadic failures that are unrelated for the PR in question - that means the remote CI is not usable for practical purposes. Problems:

it may still be that not all failures are temporarily decorated with xfail and skip properly
I think there is a even race condition in tests for the server, which results in sporadic failures, especially if multiple CI runs happen in parallel (which happens in every PR since there is a test matrix)

I am not sure how to address this best long-term; short-term I would suggest to simply skip all the flaky tests.

Mid-term, I think the server tests need to be decoupled from openml-python SDK tests, i.e., replaced by three things:

mock backends for simple API tests
a local server option that runs on the VM. This can follow the extension pattern introduced in the v1-v2 migration, see https://github.com/openml/openml-python/issues/1575
tests for the server API decoupled from tests from openml-python, possibly on a CRON and controlling for race conditions

主要语言: Python
星标: 361
派生: 296
PR 合并指标: 30 天内没有已合并 PR