combine() crashes with DataError (SQLITE_CANTOPEN) when a parallel data file vanishes mid-combine (regression in 7.17.1)

Commencez par CoverageData.read() dans coverage/sqldata.py et le flux combine_parallel_data() dans coverage/data.py ; l’issue inclut un cas reproductible déterministe et des tests de régression pour tests/test_data.py dans CoverageDataFilesTest. Vérifiez comment sont gérés les fichiers en lecture seule manquants et exécutez les deux tests de régression décrits. Le résultat attendu est qu’un avertissement signale la disparition d’un fichier, qui est comptabilisée comme une erreur tandis que les données restantes sont combinées, et qu’une lecture directe d’un fichier en lecture seule manquant lève DataError.

Describe the bug

Since 7.17.1, combine_parallel_data() (and so Coverage.combine() / coverage combine) crashes with an uncaught DataError (SQLITE_CANTOPEN) when a parallel data file disappears between being listed and being opened. 7.17.0 tolerated this. The 7.17.2 fix for #2356 handled the empty-schema and old-SQLite cases, but not a file that has vanished.

We hit it intermittently in CI: tests start multiprocessing children (spawn context) that are SIGTERMed at teardown, and pytest-cov calls cov.combine() at session end. A child that is still shutting down finishes CoverageData.write(), which renames its data file from .coverage.<host>.pid<N>.<rand> to .coverage.<host>.pid<N>.<rand>.H<hash>h, while the parent is combining. The parent has already listed the unhashed name, so it later tries to open a file that no longer exists. The failing filename in the traceback below has no .H<hash>h suffix, so it is an in-progress file.

The same failure happens on 7.17.1 (the update frame is sqldata.py line 772 there) and on 7.17.2.

To Reproduce

Environment:

• Linux, Python 3.11 (CI) and Python 3.14 (local reproduction)
• coverage 7.17.1 and 7.17.2 (fails); 7.17.0 (works)
• pytest 7.4.4, pytest-cov 4.1.0

Deterministic reproducer. It simulates the subprocess's rename by patching CoverageData.read so the in-progress file is renamed right after combine has listed it:

import os, sys, tempfile
import coverage
from coverage.data import combine_parallel_data
from coverage.sqldata import CoverageData

os.chdir(tempfile.mkdtemp())
# a finished parallel data file, plus an in-progress one that its process renames away mid-combine
d = CoverageData('.coverage.host.pid1.AAAA'); d.add_lines({'/x.py': [1, 2]}); d.write()
open('.coverage.host.pid2.BBBB', 'wb').close()

orig = CoverageData.read
def read(self):
    # simulate the subprocess finishing its write (rename to the hashed name) right after combine listed the file
    if self._filename.endswith('BBBB') and os.path.exists(self._filename):
        os.rename(self._filename, self._filename + '.Hdeadbeefh')
    return orig(self)
CoverageData.read = read

out = CoverageData('.coverage')
try:
    combine_parallel_data(out, data_paths=['.'])
    print(coverage.__version__, 'OK', sorted(os.listdir('.')))
except Exception as e:
    print(coverage.__version__, 'FAIL', type(e).__name__, e)

Output:

7.17.0 OK ['.coverage', '.coverage.host.pid2.BBBB.Hdeadbeefh']
7.17.2 FAIL DataError Couldn't use data file '/tmp/tmpeptdz4_x/.coverage.host.pid2.BBBB': unable to open database file: (SQLITE_CANTOPEN)

Traceback from CI (pytest-cov 4.1.0, coverage 7.17.2; pluggy frames trimmed):

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File ".../site-packages/coverage/sqlitedb.py", line 119, in _connect
INTERNALERROR>     self.con = sqlite3.connect(filename, check_same_thread=False, uri=True)
INTERNALERROR> sqlite3.OperationalError: unable to open database file
INTERNALERROR>
INTERNALERROR> The above exception was the direct cause of the following exception:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File ".../site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File ".../site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   [pluggy frames]
INTERNALERROR>   File ".../site-packages/pytest_cov/plugin.py", line 298, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File ".../site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>   File ".../site-packages/pytest_cov/engine.py", line 254, in finish
INTERNALERROR>     self.cov.combine()
INTERNALERROR>   File ".../site-packages/coverage/control.py", line 916, in combine
INTERNALERROR>     combine_parallel_data(
INTERNALERROR>   File ".../site-packages/coverage/data.py", line 218, in combine_parallel_data
INTERNALERROR>     data.update(new_data, map_path=map_path)
INTERNALERROR>   File ".../site-packages/coverage/sqldata.py", line 776, in update
INTERNALERROR>     with other_data._connect():
INTERNALERROR>   File ".../site-packages/coverage/sqldata.py", line 423, in _connect
INTERNALERROR>     self._open_db()
INTERNALERROR>   File ".../site-packages/coverage/sqldata.py", line 337, in _open_db
INTERNALERROR>     self._read_db()
INTERNALERROR>   File ".../site-packages/coverage/sqldata.py", line 341, in _read_db
INTERNALERROR>     with self._dbs[threading.get_ident()] as db:
INTERNALERROR>   File ".../site-packages/coverage/sqlitedb.py", line 157, in __enter__
INTERNALERROR>     self._connect()
INTERNALERROR>   File ".../site-packages/coverage/sqlitedb.py", line 121, in _connect
INTERNALERROR>     self._raise_data_error(f"Couldn't use data file {self.filename!r}", exc)
INTERNALERROR>   File ".../site-packages/coverage/sqlitedb.py", line 96, in _raise_data_error
INTERNALERROR>     raise DataError(msg) from exc
INTERNALERROR> coverage.exceptions.DataError: Couldn't use data file '/home/circleci/repo/.coverage.807f0b344fbd.pid468.XpTiFX5x': unable to open database file: (SQLITE_CANTOPEN)

Expected behavior

As in 7.17.0, and as combine_parallel_data() already does for other unusable files: report the file with a warning, count it as errored, and finish combining the rest. DataFileClassifier.classify() already anticipates this case: when it can't hash a file it returns "combine" with the comment "Probably it will fail later, but that error will be handled." For a missing file, that error isn't handled.

Cause

In combine_parallel_data() only the construction and read() are inside the try:

try:
    new_data = CoverageData(f, debug=data._debug, read_only=True)
    new_data.read()
except CoverageException as exc:
    ...  # warn, count as errored, keep the file
else:
    data.update(new_data, map_path=map_path)

CoverageData.read() is if os.path.exists(self._filename): with self._connect(): ..., so for a missing file it does nothing and nothing raises inside the try. update() then runs with other_data._connect(): pass, which reaches SqliteDb._connect(). Because the input is read_only, that opens file:...?mode=ro&immutable=1, and SQLite can't open a missing file read-only. The resulting DataError is raised from the else: branch and aborts the combine.

In 7.17.0 the input was opened read-write, so SQLite created an empty database, _read_db hit "no such table: coverage_schema", _init_db initialized it, and it was combined as empty data and deleted.

Proposed fix

Make read() fail for a missing read-only file, so the existing except in combine_parallel_data() reports it:

--- a/coverage/sqldata.py
+++ b/coverage/sqldata.py
@@ -962,6 +962,10 @@ class CoverageData:
         if os.path.exists(self._filename):
             with self._connect():
                 self._have_used = True
+        elif self._read_only:
+            # A read-only data file can't be created, so a missing one is an
+            # error now, not later when it's used.
+            raise DataError(f"Couldn't use data file {self._filename!r}: it doesn't exist")
 
     def _data_hash(self) -> str:
         """The short hash of the data so far, for the data file name."""

The only other read_only=True caller, debug_data_file(), checks os.path.exists() before calling read(), so it is unaffected.

With this patch applied to the 7.17.2 sdist, the reproducer prints 7.17.2 OK ['.coverage', '.coverage.host.pid2.BBBB.Hdeadbeefh'], and Coverage.combine() issues a Couldn't use data file '...': it doesn't exist warning instead of crashing.

Regression tests for tests/test_data.py (CoverageDataFilesTest). Both fail on 7.17.2 and pass with the patch:

+    def test_combining_with_vanishing_file(self) -> None:
+        # A process still finishing its write renames its data file after
+        # combine has listed it.  The vanished file is reported, not fatal.
+        covdata1 = DebugCoverageData(suffix="1")
+        covdata1.add_lines(LINES_1)
+        covdata1.write()
+        self.make_file(".coverage.2", "")
+
+        real_read = CoverageData.read
+
+        def renaming_read(self: CoverageData) -> None:
+            fname = self.data_filename()
+            if fname.endswith(".coverage.2") and os.path.exists(fname):
+                os.rename(fname, fname + ".Hdeadbeefh")
+            real_read(self)
+
+        warnings: list[str] = []
+        covdata2 = DebugCoverageData()
+        covdata2._warn = warnings.append
+        with mock.patch.object(CoverageData, "read", renaming_read):
+            combine_parallel_data(covdata2)
+
+        assert_line_counts(covdata2, SUMMARY_1)
+        assert len(warnings) == 1
+        assert re.search(r"Couldn't use data file '.*\.coverage\.2': it doesn't exist", warnings[0])
+        self.assert_doesnt_exist(".coverage.1")
+        self.assert_exists(".coverage.2.Hdeadbeefh")
+
+    def test_reading_missing_read_only_file(self) -> None:
+        covdata = DebugCoverageData("nope", read_only=True)
+        with pytest.raises(DataError, match=r"Couldn't use data file '.*nope': it doesn't exist"):
+            covdata.read()

Other options and trade-offs:

• Moving data.update() into the try would also catch this, but it would turn every DataError from update() into a warning. That includes the cant-combine error for mixed line and branch data, which test_combining_cant_mix_lines_and_arcs expects to propagate, and the file-tracer conflict error. For the vanished-file case it would be safe: update() opens other_data (with other_data._connect(): pass) and ATTACHes it before it writes anything to the target. _start_using() only makes sure the target database exists. Errors raised after the inserts begin are another matter: the target runs with journal_mode=off, so I wouldn't rely on a rollback to undo a partial merge.
• A narrower window remains. If the file disappears after read() has checked it but before update() opens or ATTACHes it, the same uncaught DataError is possible. That window is a few statements long. The window this patch closes covers the whole combine loop up to that file's turn. Closing the narrow window as well would mean catching open/ATTACH failures inside update(), or catching only those errors (not cant-combine) around the update() call.

Related: #2356.

---

This issue was triaged with the help of Claude Code: it diagnosed the CI failure, wrote and ran the reproducer, and drafted the proposed patch and tests. I reviewed the results before filing.