Debug advice on intermittent FileNotFound error when collecting distributed coverage data in Jenkins

Begin by reproducing the Jenkins failure with the listed Python, pytest, pytest-cov, pytest-xdist, and coverage versions, then inspect the stack-trace path through pytest_cov/plugin.py:297, pytest_cov/engine.py:338, and coverage/control.py:790. The issue provides no repository file, test, or requested code change, so completion would first require identifying a reproducible cause and agreeing on the expected fix or diagnostic output.

Summary

We have some random Jenkins build failures. Some pytest runs fail (after all actual tests succeed) with an INTERNALERROR, when coverage data is collected from the pytest-xdist workers. I don't know how to proceed, or what I can do to get more debug output. Thanks in advance!

Expected vs actual result

Some times the result is as expected (coverage data is collected just fine), some times it is not.

Reproducer

Versions

We use Python 3.9.10 on GNU/Linux. Excerpt from our setup.cfg:

[options.extras_require]
test =
    coverage >= 7.1, < 7.2
    pytest >= 7.2, < 7.3
    pytest-metadata != 2.0.0
    pytest-cov >= 4.0, < 4.1
    pytest-html >= 3.2, < 3.3
    pytest-sugar >= 0.9, < 0.10
    pytest-xdist >= 3.1, < 3.2

Config

Excerpt from our (anonymized) pyproject.toml

[tool.pytest.ini_options]
addopts = "-q --self-contained-html --css=tests/fixtures/report.css"
markers = [
  "unit: marks tests as unit tests. Will be added automatically if not integration, verification or validation test.",
  "integration: marks tests as integration test.",
  "verification: marks tests as verification test.",
  "validation: marks tests as validation test.",
  "slow: marks tests as slow (runtime > 5min).",
  "external: marks tests that require external inputs.",
]
testpaths = [
  "tests",
]

[tool.coverage.run]
omit = [
  "src/proj/__init__.py",
  "src/proj/_version.py",
]
source = [
  "src/proj/",
]

[tool.coverage.report]
exclude_lines = [
  "pragma: no cover",
  "def __repr__",
  "def __str__",
]

Code

Unfortunately, I can't provide code. But here's the (anonymized) stack trace from the error:

INTERNALERROR> config = <_pytest.config.Config object at 0x7fb707e12340>
INTERNALERROR> doit = <function _main at 0x7fb708b32dc0>
INTERNALERROR> 
INTERNALERROR>     def wrap_session(
INTERNALERROR>         config: Config, doit: Callable[[Config, "Session"], Optional[Union[int, ExitCode]]]
INTERNALERROR>     ) -> Union[int, ExitCode]:
INTERNALERROR>         """Skeleton command line program."""
INTERNALERROR>         session = Session.from_config(config)
INTERNALERROR>         session.exitstatus = ExitCode.OK
INTERNALERROR>         initstate = 0
INTERNALERROR>         try:
INTERNALERROR>             try:
INTERNALERROR>                 config._do_configure()
INTERNALERROR>                 initstate = 1
INTERNALERROR>                 config.hook.pytest_sessionstart(session=session)
INTERNALERROR>                 initstate = 2
INTERNALERROR> >               session.exitstatus = doit(config, session) or 0
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/_pytest/main.py:270: 
INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
INTERNALERROR> 
INTERNALERROR> config = <_pytest.config.Config object at 0x7fb707e12340>
INTERNALERROR> session = <Session job exitstatus=<ExitCode.INTERNAL_ERROR: 3> testsfailed=0 testscollected=949>
INTERNALERROR> 
INTERNALERROR>     def _main(config: Config, session: "Session") -> Optional[Union[int, ExitCode]]:
INTERNALERROR>         """Default command line protocol for initialization, session,
INTERNALERROR>         running tests and reporting."""
INTERNALERROR>         config.hook.pytest_collection(session=session)
INTERNALERROR> >       config.hook.pytest_runtestloop(session=session)
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/_pytest/main.py:324: 
INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
INTERNALERROR> 
INTERNALERROR> self = <_HookCaller 'pytest_runtestloop'>, args = ()
INTERNALERROR> kwargs = {'session': <Session job exitstatus=<ExitCode.INTERNAL_ERROR: 3> testsfailed=0 testscollected=949>}
INTERNALERROR> argname = 'session', firstresult = True
INTERNALERROR> 
INTERNALERROR>     def __call__(self, *args, **kwargs):
INTERNALERROR>         if args:
INTERNALERROR>             raise TypeError("hook calling supports only keyword arguments")
INTERNALERROR>         assert not self.is_historic()
INTERNALERROR>     
INTERNALERROR>         # This is written to avoid expensive operations when not needed.
INTERNALERROR>         if self.spec:
INTERNALERROR>             for argname in self.spec.argnames:
INTERNALERROR>                 if argname not in kwargs:
INTERNALERROR>                     notincall = tuple(set(self.spec.argnames) - kwargs.keys())
INTERNALERROR>                     warnings.warn(
INTERNALERROR>                         "Argument(s) {} which are declared in the hookspec "
INTERNALERROR>                         "can not be found in this hook call".format(notincall),
INTERNALERROR>                         stacklevel=2,
INTERNALERROR>                     )
INTERNALERROR>                     break
INTERNALERROR>     
INTERNALERROR>             firstresult = self.spec.opts.get("firstresult")
INTERNALERROR>         else:
INTERNALERROR>             firstresult = False
INTERNALERROR>     
INTERNALERROR> >       return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/pluggy/_hooks.py:265: 
INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
INTERNALERROR> 
INTERNALERROR> self = <_pytest.config.PytestPluginManager object at 0x7fb712d80730>
INTERNALERROR> hook_name = 'pytest_runtestloop'
INTERNALERROR> methods = [<HookImpl plugin_name='main', plugin=<module '_pytest.main' from '/opt/data/jenkins/project/workspace/job...b7078ea6d0>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object at 0x7fb6a435c700>>]
INTERNALERROR> kwargs = {'session': <Session job exitstatus=<ExitCode.INTERNAL_ERROR: 3> testsfailed=0 testscollected=949>}
INTERNALERROR> firstresult = True
INTERNALERROR> 
INTERNALERROR>     def _hookexec(self, hook_name, methods, kwargs, firstresult):
INTERNALERROR>         # called from all hookcaller instances.
INTERNALERROR>         # enable_tracing will set its own wrapping function at self._inner_hookexec
INTERNALERROR> >       return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/pluggy/_manager.py:80: 
INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
INTERNALERROR> 
INTERNALERROR> self = <pytest_cov.plugin.CovPlugin object at 0x7fb7078ea6d0>
INTERNALERROR> session = <Session job exitstatus=<ExitCode.INTERNAL_ERROR: 3> testsfailed=0 testscollected=949>
INTERNALERROR> 
INTERNALERROR>     @pytest.hookimpl(hookwrapper=True)
INTERNALERROR>     def pytest_runtestloop(self, session):
INTERNALERROR>         yield
INTERNALERROR>     
INTERNALERROR>         if self._disabled:
INTERNALERROR>             return
INTERNALERROR>     
INTERNALERROR>         compat_session = compat.SessionWrapper(session)
INTERNALERROR>     
INTERNALERROR>         self.failed = bool(compat_session.testsfailed)
INTERNALERROR>         if self.cov_controller is not None:
INTERNALERROR> >           self.cov_controller.finish()
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/pytest_cov/plugin.py:297: 
INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
INTERNALERROR> 
INTERNALERROR> self = <pytest_cov.engine.DistMaster object at 0x7fb7078b8160>, args = ()
INTERNALERROR> kwargs = {}
INTERNALERROR> original_cwd = '/opt/data/jenkins/project/workspace/job'
INTERNALERROR> 
INTERNALERROR>     @functools.wraps(meth)
INTERNALERROR>     def ensure_topdir_wrapper(self, *args, **kwargs):
INTERNALERROR>         try:
INTERNALERROR>             original_cwd = os.getcwd()
INTERNALERROR>         except OSError:
INTERNALERROR>             # Looks like it's gone, this is non-ideal because a side-effect will
INTERNALERROR>             # be introduced in the tests here but we can't do anything about it.
INTERNALERROR>             original_cwd = None
INTERNALERROR>         os.chdir(self.topdir)
INTERNALERROR>         try:
INTERNALERROR> >           return meth(self, *args, **kwargs)
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/pytest_cov/engine.py:44: 
INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
INTERNALERROR> 
INTERNALERROR> self = <pytest_cov.engine.DistMaster object at 0x7fb7078b8160>
INTERNALERROR> 
INTERNALERROR>     @_ensure_topdir
INTERNALERROR>     def finish(self):
INTERNALERROR>         """Combines coverage data and sets the list of coverage objects to report on."""
INTERNALERROR>     
INTERNALERROR>         # Combine all the suffix files into the data file.
INTERNALERROR> >       self.cov.stop()
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/pytest_cov/engine.py:338: 
INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
INTERNALERROR> 
INTERNALERROR> self = <coverage.control.Coverage object at 0x7fb7078b8220>, data_paths = None
INTERNALERROR> strict = False, keep = False
INTERNALERROR> 
INTERNALERROR>     def combine(
INTERNALERROR>         self,
INTERNALERROR>         data_paths: Optional[Iterable[str]] = None,
INTERNALERROR>         strict: bool = False,
INTERNALERROR>         keep: bool = False
INTERNALERROR>     ) -> None:
INTERNALERROR>         """Combine together a number of similarly-named coverage data files.
INTERNALERROR>     
INTERNALERROR>         All coverage data files whose name starts with `data_file` (from the
INTERNALERROR>         coverage() constructor) will be read, and combined together into the
INTERNALERROR>         current measurements.
INTERNALERROR>     
INTERNALERROR>         `data_paths` is a list of files or directories from which data should
INTERNALERROR>         be combined. If no list is passed, then the data files from the
INTERNALERROR>         directory indicated by the current data file (probably the current
INTERNALERROR>         directory) will be combined.
INTERNALERROR>     
INTERNALERROR>         If `strict` is true, then it is an error to attempt to combine when
INTERNALERROR>         there are no data files to combine.
INTERNALERROR>     
INTERNALERROR>         If `keep` is true, then original input data files won't be deleted.
INTERNALERROR>     
INTERNALERROR>         .. versionadded:: 4.0
INTERNALERROR>             The `data_paths` parameter.
INTERNALERROR>     
INTERNALERROR>         .. versionadded:: 4.3
INTERNALERROR>             The `strict` parameter.
INTERNALERROR>     
INTERNALERROR>         .. versionadded: 5.5
INTERNALERROR>             The `keep` parameter.
INTERNALERROR>         """
INTERNALERROR>         self._init()
INTERNALERROR>         self._init_data(suffix=None)
INTERNALERROR>         self._post_init()
INTERNALERROR>         self.get_data()
INTERNALERROR>     
INTERNALERROR> >       combine_parallel_data(
INTERNALERROR>             self._data,
INTERNALERROR>             aliases=self._make_aliases(),
INTERNALERROR>             data_paths=data_paths,
INTERNALERROR>             strict=strict,
INTERNALERROR>             keep=keep,
INTERNALERROR>             message=self._message,
INTERNALERROR>         )
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/coverage/control.py:790: 
INTERNALERROR> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
INTERNALERROR> 
INTERNALERROR> data = <CoverageData @0x7fb6f419b970 _no_disk=False _basename='/opt/data/jenkins/project/workspace/job..._have_used=True _has_lines=False _has_arcs=True _current_context=None _current_context_id=None _query_context_ids=None>
INTERNALERROR> aliases = <coverage.files.PathAliases object at 0x7fb6f419b3d0>
INTERNALERROR> data_paths = None, strict = False, keep = False
INTERNALERROR> message = <bound method Coverage._message of <coverage.control.Coverage object at 0x7fb7078b8220>>
INTERNALERROR> 
INTERNALERROR>     def combine_parallel_data(
INTERNALERROR>         data: CoverageData,
INTERNALERROR>         aliases: Optional[PathAliases] = None,
INTERNALERROR>         data_paths: Optional[Iterable[str]] = None,
INTERNALERROR>         strict: bool = False,
INTERNALERROR>         keep: bool = False,
INTERNALERROR>         message: Optional[Callable[[str], None]] = None,
INTERNALERROR>     ) -> None:
INTERNALERROR>         """Combine a number of data files together.
INTERNALERROR>     
INTERNALERROR>         `data` is a CoverageData.
INTERNALERROR>     
INTERNALERROR>         Treat `data.filename` as a file prefix, and combine the data from all
INTERNALERROR>         of the data files starting with that prefix plus a dot.
INTERNALERROR>     
INTERNALERROR>         If `aliases` is provided, it's a `PathAliases` object that is used to
INTERNALERROR>         re-map paths to match the local machine's.
INTERNALERROR>     
INTERNALERROR>         If `data_paths` is provided, it is a list of directories or files to
INTERNALERROR>         combine.  Directories are searched for files that start with
INTERNALERROR>         `data.filename` plus dot as a prefix, and those files are combined.
INTERNALERROR>     
INTERNALERROR>         If `data_paths` is not provided, then the directory portion of
INTERNALERROR>         `data.filename` is used as the directory to search for data files.
INTERNALERROR>     
INTERNALERROR>         Unless `keep` is True every data file found and combined is then deleted
INTERNALERROR>         from disk. If a file cannot be read, a warning will be issued, and the
INTERNALERROR>         file will not be deleted.
INTERNALERROR>     
INTERNALERROR>         If `strict` is true, and no files are found to combine, an error is
INTERNALERROR>         raised.
INTERNALERROR>     
INTERNALERROR>         `message` is a function to use for printing messages to the user.
INTERNALERROR>     
INTERNALERROR>         """
INTERNALERROR>         files_to_combine = combinable_files(data.base_filename(), data_paths)
INTERNALERROR>     
INTERNALERROR>         if strict and not files_to_combine:
INTERNALERROR>             raise NoDataError("No data to combine")
INTERNALERROR>     
INTERNALERROR>         file_hashes = set()
INTERNALERROR>         combined_any = False
INTERNALERROR>     
INTERNALERROR>         for f in files_to_combine:
INTERNALERROR>             if f == data.data_filename():
INTERNALERROR>                 # Sometimes we are combining into a file which is one of the
INTERNALERROR>                 # parallel files.  Skip that file.
INTERNALERROR>                 if data._debug.should('dataio'):
INTERNALERROR>                     data._debug.write(f"Skipping combining ourself: {f!r}")
INTERNALERROR>                 continue
INTERNALERROR>     
INTERNALERROR>             try:
INTERNALERROR>                 rel_file_name = os.path.relpath(f)
INTERNALERROR>             except ValueError:
INTERNALERROR>                 # ValueError can be raised under Windows when os.getcwd() returns a
INTERNALERROR>                 # folder from a different drive than the drive of f, in which case
INTERNALERROR>                 # we print the original value of f instead of its relative path
INTERNALERROR>                 rel_file_name = f
INTERNALERROR>     
INTERNALERROR> >           with open(f, "rb") as fobj:
INTERNALERROR> E           FileNotFoundError: [Errno 2] No such file or directory: '/opt/data/jenkins/project/workspace/job/.coverage.myhost.25783.340214'
INTERNALERROR> 
INTERNALERROR> .env/lib/python3.9/site-packages/coverage/data.py:148: FileNotFoundError

What has been tried to solve the problem

Adding --full-trace option to pytest in our Jenkinsfile.