Test Runner for Java ignores @Order annotations on test classes

Open
#1,748 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
java, typescript, vscode
Domain
devtools, testing-qa

Research direction

Start with the VSCode Testing view execution path and compare it with the Gradle test run, using related issues #1740 and #1756 as context. Verify whether class-level @Order and @EnabledIf behavior is preserved; done means ordered classes and methods run consistently, or the supported runner-version override is documented.

Written by the indexing model from the issue text.

Description

ai-triaged bug gradle high-priority junit test-execution

Symptom: JUnit 5 test classes which reliably run in the correct order with gradlew test run in the wrong order in the VSCode IDE.

System: Windows 10, VSCode 1.96.3, Gradle 8.12, Spring Boot 3.4.1, Java 21.04, JUnit 5.11.4

Use case: In a Spring Boot client-server Gradle project, the integration tests in the client execute against the running server. The server-side database state built up by earlier integration tests is required by the later integration tests and thus the tests must be executed in a specific order. Accordingly, the test classes and methods are annotated with @Order and the configuration property junit.jupiter.testclass.order.default = org.junit.jupiter.api.ClassOrderer$OrderAnnotation is set. Executed under Gradle the test classes and methods all run in the designated order. However, in the VSCode Testing view contributed by this extension, the test classes get executed in the wrong order and some test methods don't run at all because they use the @EnabledIf annotation to check whether requisite client-side state (reflecting server-side state) has been collected by tests that are supposed to have run already but which in fact haven't yet run.

I've scoured the docs, searched the web and asked several AI chatbots about this but haven't found any solutions that work.

Hypothesis: the test runner is embedded in the extension (v. 0.43.0) and is of too early a version to support test class ordering (~JUnit 5.8, I believe).

Is it possible to override the test runner version?

Related issues: https://github.com/microsoft/vscode-java-test/issues/1740, #1756

Dominant language
TypeScript
Stars
340
Forks
173
Avg merge
1d 2h
Merged PRs (30d)
17

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/vscode-java-test

All issues in microsoft/vscode-java-test

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.