Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[BUG] resolve failing and flaky tests in CI, and race conditions in tests using server

Aberta
  • Algo errado?
#1,586 4 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
25/100
Tipo de issue
Bug
Clareza
Precisa de esclarecimento
Status de atividade
Pouca atividade
Stack de tecnologia
python
Domínio
backend, ci-cd, testing

Direção de pesquisa

Start by reproducing the reported flaky failures in remote CI and inspect the server tests for shared-state or parallel-run races. Review the extension pattern from issue 1575 and compare the proposed mock, local-server, and decoupled server-test approaches. Done means the CI test matrix is reliable and server tests no longer fail sporadically when runs overlap.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

bug testing

Currently, the remote CI is not reliable and will result in faliures and sporadic failures that are unrelated for the PR in question - that means the remote CI is not usable for practical purposes. Problems:

  • it may still be that not all failures are temporarily decorated with xfail and skip properly
  • I think there is a even race condition in tests for the server, which results in sporadic failures, especially if multiple CI runs happen in parallel (which happens in every PR since there is a test matrix)

I am not sure how to address this best long-term; short-term I would suggest to simply skip all the flaky tests.

Mid-term, I think the server tests need to be decoupled from openml-python SDK tests, i.e., replaced by three things:

  • mock backends for simple API tests
  • a local server option that runs on the VM. This can follow the extension pattern introduced in the v1-v2 migration, see https://github.com/openml/openml-python/issues/1575
  • tests for the server API decoupled from tests from openml-python, possibly on a CRON and controlling for race conditions
Linguagem predominante
Python
Estrelas
366
Forks
308
Merge médio
1d 16h
PRs com merge (30d)
7

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de openml/openml-python

Todas as issues de openml/openml-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.