WPT CLI: Errors raised during environment initialization cause invalid output
#9,481 opened on 2018年2月12日
説明
If the WPT CLI encounters an error when initializing an "environment," an empty file is created for the --log-wptreport option. Because the empty string is not valid JSON, users must take special care when interpreting the contents of that file.
This is observable when using the Sauce Labs service to run WPT, and the Sauce Connect tunnel fails to open. These conditions can be consistently reproduced on Unix-like systems with the following command:
./wpt run --sauce-platform linux \
--sauce-tunnel-id fake \
--sauce-user fake \
--sauce-key fake \
--sauce-connect-binary /bin/false \
--log-wptreport empty.json \
sauce:firefox:8
This issue is not limited to the use of the Sauce Labs service. For example, errors that occur during font installation will also yield invalid output:
./wpt run --install-fonts \
--font-dir /non-existent-directory \
--log-wptreport empty.json \
firefox
From a cursory investigation, it looks like the file is being created eagerly by the MozLog utility.
I can think of two solutions:
- Remove that file whenever the tests do not run to completion. This may not be feasible if there isn't a central location to detect the type of failures that may cause this behavior
- Initially write valid JSON to that file, preferably a structure that aligns with the expected structure but unambiguously communicates the issue. For example:
{"results":[],"incomplete":true}. This ensures that the output file consistently describes the state of the test running process, and it may be later extended to support partial result reporting in the event of other runtime failures.