web-platform-tests/wpt

WPT CLI: Errors raised during environment initialization cause invalid output

Open

#9.481 aberto em 12 de fev. de 2018

Ver no GitHub
 (7 comments) (1 reaction) (0 assignees)HTML (2.898 forks)batch import
good first issueinfrapriority:backlog

Métricas do repositório

Stars
 (4.581 stars)
Métricas de merge de PR
 (Mesclagem média 5d 23h) (682 fundiu PRs em 30d)

Description

If the WPT CLI encounters an error when initializing an "environment," an empty file is created for the --log-wptreport option. Because the empty string is not valid JSON, users must take special care when interpreting the contents of that file.

This is observable when using the Sauce Labs service to run WPT, and the Sauce Connect tunnel fails to open. These conditions can be consistently reproduced on Unix-like systems with the following command:

./wpt run --sauce-platform linux \
  --sauce-tunnel-id fake \
  --sauce-user fake \
  --sauce-key fake \
  --sauce-connect-binary /bin/false \
  --log-wptreport empty.json \
  sauce:firefox:8

This issue is not limited to the use of the Sauce Labs service. For example, errors that occur during font installation will also yield invalid output:

./wpt run --install-fonts \
  --font-dir /non-existent-directory \
  --log-wptreport empty.json \
  firefox

From a cursory investigation, it looks like the file is being created eagerly by the MozLog utility.

I can think of two solutions:

  • Remove that file whenever the tests do not run to completion. This may not be feasible if there isn't a central location to detect the type of failures that may cause this behavior
  • Initially write valid JSON to that file, preferably a structure that aligns with the expected structure but unambiguously communicates the issue. For example: {"results":[],"incomplete":true}. This ensures that the output file consistently describes the state of the test running process, and it may be later extended to support partial result reporting in the event of other runtime failures.

Guia do colaborador