Caching behavior related to PATH and use_default_shell_env are problematic
#18,809 opened on Jun 29, 2023
Repository metrics
- Stars
- (25,384 stars)
- PR merge metrics
- (Avg merge 22d 20h) (77 merged PRs in 30d)
Description
Currently:
- The Bazel action environment (which also affects its cache key) inherit
PATHandLD_LIBRARY_PATHby default. If--incompatible_strict_action_envis specified, then a defaultPATHis used andLD_LIBRARY_PATHis not inherited. (Or these environment variables can be explicitly overridden via--action_env.) - The
use_default_shell_envoption defaults toFalse, which means by default commands are run with noPATH. Ifuse_default_shell_env=Trueare specified, then the normalPATHandLD_LIBRARY_PATHvalues for theaction_envare used.
This leads to the following problems:
- When invoking Bazel from a Python setup.py script in order to build a Python extension with Bazel, as is done by tensorstore, the
PATHmay include temporary virtualenv directories created by pip. Even when usingpip install -e .to install for local development purposes, pip will by default use an isolated build environment (which results in these temporary directories being added to thePATHin order to isolate build dependencies. This means that neither local nor remote caching works. - If we specify
--incompatible_strict_action_env, thenPATHis no longer passed to the action, and no longer affects the cache key, but this breaks building in environments/with toolchains that requirePATHand/orLD_LIBRARY_PATHbe set. For example, on Windows we may wish to use a self-contained version of MSVC or mingw where we haven't copied the various runtime DLLs to the Windows system directory. As a result, any binaries built will only run if thePATHis preserved so that they can locate the DLLs that they require (Windows does not have the equivalent of rpath). (See https://github.com/bazelbuild/bazel/issues/15059) - Even if we don't specify
--incompatible_strict_action_env, a lot of rules still don't work with toolchains that requirePATHto be set, because they invoke executables without settinguse_default_shell_env=True. Since the rule author doesn't know the details of the toolchain or system environment, they cannot know that the defaultuse_default_shell_env=Falsewill be problematic.
I propose the following changes:
- Introduce
--exclude_action_env_from_cache=Xto exclude a given action environment variable (if inherited or specified on the command-line, but not if specified in the rule directly) from the cache key. (Previously proposed here some years ago https://groups.google.com/g/bazel-discuss/c/AZek-OR1t7A) - Introduce
--incompatible_exclude_path_from_action_cache_key. If set, this is equivalent to specifying--exclude_action_env_from_cache={PATH,LD_LIBRARY_PATH}. - Introduce
--incompatible_use_default_shell_envwhich setsuse_default_shell_env=Trueby default. This would require https://github.com/bazelbuild/bazel/pull/18235 to be practical.
As noted in https://groups.google.com/g/bazel-discuss/c/AZek-OR1t7A, including PATH in the action cache key does not ensure hermeticity anyway except in special cases such as when using nix, where the PATH is guaranteed to change if any of the executables change. In this rare case where including PATH in the cache key is helpful, --incompatible_exclude_path_from_action_cache_key=false can be specified by the user.
In https://github.com/bazelbuild/bazel/issues/15059 an alternative to defaulting use_default_shell_env=True to solve the issues with mingw is proposed, that the toolchain should somehow add all of the runtime DLLs as runfiles. Note that the issue is not specific to mingw, though --- we may wish to build with a self-contained copy of MSVC (in fact we do this in the tensorstore CI build) and the same issue applies there, so this would basically be needed for all toolchains. This may work, but in general by breaking normal expectations of how the PATH should be handled, creates additional trouble for users. For example, if any other binaries used by actions need DLLs, those DLLs have to be copied to the same directory as the binary on Windows if PATH cannot be relied upon.