github/vscode-codeql

Test execution timeout it not properly detected

開放

#1,496 建立於 2022年9月5日

 (2 則留言) (0 個反應) (0 位負責人)TypeScript (236 個分叉)auto 404
VSCodebuggood first issue

倉庫指標

星標
 (530 顆星)
PR 合併指標
 (PR 指標待抓取)

描述

Describe the bug When test execution times out (after 5 minutes by default), the extension does not seem to notice that properly:

  • The "Testing" tab still shows the test as running
  • The timeout error appears (only) in the .actual file instead of the "CodeQL Tests" output (similar to #819)

Version Extension:

CodeQL extension version: 1.6.12 
CodeQL CLI version: 2.10.4 
Platform: win32 x64

VS Code:

Version: 1.71.0 (user setup)
Commit: 784b0177c56c607789f9638da7b6bf3230d47a8c
Date: 2022-09-01T07:36:10.600Z
Electron: 19.0.12
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.19043
Sandboxed: No

To reproduce

  1. Open the extension settings
  2. Edit "Additional Test Arguments" and add "--timeout=1"
  3. Run a test :x: The extension does not properly handle the test failure

(remember to revert your settings changes)

Expected behavior

  • The extension should properly detect that the test failed
  • The timeout error message should appear in the "CodeQL Tests" output and not in the .actual file

Additional context The .actual file contains:

Timeout (5m0s) in api-visibility-issues#09cf2679::getATypeExposingElement#6#fffffff

(with api-visibility-issues being the query name)

Running the test with codeql test run --format=json yields the following response (might helpful to adjust the parsing logic in the extension?):

{
  "test" : "<test-dir>/api-visibility-issues.qlref",
  "pass" : false,
  "failureStage" : "EXECUTION",
  "messages" : [ ],
  "evaluationMs" : 305698,
  "expected" : "<test-dir>/api-visibility-issues.expected",
  "actual" : "<test-dir>/api-visibility-issues.actual",
  "diff" : [
    "--- expected",
    "+++ actual",
    "@@ -1,30 +1,1 @@",
    "-| ...",
    "+Timeout (5m0s) in api-visibility-issues#09cf2679::getATypeExposingElement#6#fffffff"
  ]
}

(slightly minified)

貢獻者指南