Add consistency for reporting TestNG results for data providers

Open
#787 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
java, typescript, vscode
Domain
developer-experience, testing

Research direction

Start with the default TestNG listener and the reporting path for data providers; the issue's ITest/getTestName workaround and linked IntelliJ listener provide comparison points. Reproduce the case from the issue and verify that the VS Code UI and console report consistent test counts.

Written by the indexing model from the issue text.

Description

ai-triaged bug enhancement test-reporting testng ui
Summary

Overall, great experience working with vscode and this extension thus far. Adding this feature/fixing the bug (eye of the beholder) will save a bit of time running commands in the terminal to get reliable results.

See below for a depiction of the disagreement between the user interface and console reports as well as a simple reproduce case.

I'd imagine there are many ways to approach this for full flexibility but I do think having the default test listener support this out of the box and at least have agreement in counts would be a desirable feature. I have used other IDEs and here is one way it is being approached: https://github.com/JetBrains/intellij-community/blob/2aa816fb814bb07c697a17d6a27e427e142a3633/plugins/testng_rt/src/com/intellij/rt/testng/IDEATestNGRemoteListener.java.

Note: there are ways to work around this by implementing the ITest interface per test and overriding the getTestName method but this is cumbersome as well!

System and application versions
Version: 1.37.1 (user setup)
Commit: f06011ac164ae4dc8e753a3fe7f9549844d15e35
Date: 2019-08-15T16:17:55.855Z
Electron: 4.2.7
Chrome: 69.0.3497.128
Node.js: 10.11.0
V8: 6.9.427.31-electron.0
OS: Windows_NT x64 10.0.18362
TestNG version
  <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.8</version>
            <scope>test</scope>
  </dependency>
Repro case

image

Dominant language
TypeScript
Stars
340
Forks
173
Avg merge
1d 2h
Merged PRs (30d)
17

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/vscode-java-test

All issues in microsoft/vscode-java-test

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.