webdriverio/webdriverio

[💡 Feature]: Wdio spec reporter should exclude hook failures in test case pass fail count

Open

#12 524 ouverte le 18 mars 2024

Voir sur GitHub
 (7 commentaires) (0 réactions) (0 assignés)JavaScript (6 029 stars) (1 793 forks)batch import
Idea 💡help wanted

Description

Is your feature request related to a problem?

Yes, I am using "@wdio/junit-reporter": "^8.32.4", and getting report with beforeAll Hook and afterAll Hook failed from the spec-reporter report like below. In case of success, it excludes the hooks in spec reporter. So I get 4 passed. In case of failure, it includes the hooks in spec reporter. So I get 6 failed. In success/failure, the total count should not be changed for me. This is shown in Dashboard in graphs making difficult to understand by business team, why it shows count less one day (in case of success) and other day count more (in case of failure). [edge-headless-shell 122.0.2365.52 windows #0-0] Spot - A and P [edge-headless-shell 122.0.2365.52 windows #0-0] Verify whether the user is able to search with a ID in Spot screen [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate A and P tab is Highlighted [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate RMC Menu Details [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate Export RMC Menu [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ "after all" hook [edge-headless-shell 122.0.2365.52 windows #0-0] [edge-headless-shell 122.0.2365.52 windows #0-0] Verify No records text in A and P Tab [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ "before all" hook [edge-headless-shell 122.0.2365.52 windows #0-0] ✖ Validate No Records [edge-headless-shell 122.0.2365.52 windows #0-0] [edge-headless-shell 122.0.2365.52 windows #0-0] 6 failing (27.9s)

Describe the solution you'd like.

Exclude the hooks count in total test cases passed or failed. Hooks are meant to setup initial or final state of the application/test suite. Reference to Discord channel: https://discord.com/channels/1097401827202445382/1097674067601010709/1218898644284080208

Describe alternatives you've considered.

Not sure, as Junit or HTML reports generated on the basis of default spec reporter generated, in the spec reporter itself this issue is seen.

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Guide contributeur

Stack technique
javascriptnodejs
Domaine
testing
Type d'issue
feature
DifficultéDifficulté estimée pour un nouveau contributeur, de 1 pour un très petit changement à 5 pour un travail expert.
3
Temps estiméFourchette de temps approximative pour investiguer, implémenter, tester et préparer une pull request.
1-2 days
Statut d'activitéDisponibilité apparente de l'issue : fraîche, active, ancienne, bloquée ou en attente d'un mainteneur.
active
ClartéClarté avec laquelle l'issue explique le changement attendu, les critères d'acceptation et la prochaine étape.
mostly clear
Prérequis
understanding of webdriverio reporter architecturebasic JavaScript
Accessibilité débutantScore de 1 à 100 estimant l'accessibilité de cette issue pour un premier contributeur.
45
Direction de recherche
The issue requests that hook failures (before/after all) be excluded from the test case pass/fail count in the spec reporter. The relevant code is likely in the spec reporter package (packages/wdio spec reporter). The Discord reference indicates community discussion. The fix would involve modifying the logic that counts test results, ensuring hooks are not counted as individual test cases. Look at the spec reporter's getTestCount or similar methods to understand current counting behavior. Also check the junit reporter as it may depend on the same logic. The issue has no assignee, so this is available for contribution.