make_copy() in concurrent processes fails or silently returns a wrong copy (unseeded rand() gives identical temp file names)

Open
#267 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
40/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp, python
Domain
backend, testing

Research direction

The collision starts at the temp-file name built from rand() in makeCopy() in geodiff/src/geodiff.cpp, and randomString() and the TmpFile destructor in geodiff/src/geodiffutils.cpp handle the same naming and deletion. Read those, then run the Python reproduction in the issue, which should fail on a shared /tmp and pass with --private-tmpdir. Done means concurrent make_copy() calls return correct copies on a shared TMPDIR and ctest -j passes without collisions on the changeset file.

Written by the indexing model from the issue text.

Description

Summary

GEODIFF_makeCopy() (make_copy() in Python) writes the source data to a temporary changeset whose name comes from rand(), and nothing in geodiff ever seeds rand(). Every process therefore generates the same sequence of "random" names, so separate processes running make_copy() at the same time share one temporary file (and the first to finish deletes it). In the reproduction below this makes make_copy() raise Table does not exist, or return success with a copy that differs from the source.

The same thing makes ctest -j fail intermittently on master (see the end).

Where

At commit 82fe7d8 (2.3.1 + 2):

  • geodiff.cpp#L306: TmpFile tmpFileChangeset( tmpdir( ) + "geodiff_changeset" + std::to_string( rand() ) ); in makeCopy()
  • TmpFile::~TmpFile() removes that file by name, so one process deletes the file another one is still using.
  • srand() and std::random_device are not used anywhere in the repository.
  • On glibc the first rand() of every process returns 1804289383, so a freshly started process always uses /tmp/geodiff_changeset1804289383 (confirmed with strace).
  • randomString() (geodiffutils.cpp#L655) uses the same unseeded rand() and names the temporary files of GEODIFF_createChangesetDr() (_gpkg-xxxxxx, L392, L407) and of GEODIFF_rebaseEx() (geodiff_xxxxxx_base2theirs.bin etc., L886). I only reproduced the problem with make_copy(); these two I know only from reading the code.

Reproduction

Several independent processes call make_copy() at the same moment, each on its own database, and check that each copy matches its own source. Run with pip install pygeodiff.

"""
Independent processes call make_copy() at the same moment, each on its own database.

    python repro_makecopy_race.py                   # default: all processes share /tmp
    python repro_makecopy_race.py --private-tmpdir  # control: every process has its own TMPDIR
"""

import multiprocessing as mp
import os
import sqlite3
import sys
import tempfile
from collections import Counter

import pygeodiff

ROWS, PROCESSES, ROUNDS = 100_000, 4, 10


def summary(path, table):
    """(row count, sum of ids) of a table, None if it does not exist"""
    db = sqlite3.connect(path)
    if not db.execute("SELECT 1 FROM sqlite_master WHERE name = ?", (table,)).fetchone():
        return None
    return db.execute(f"SELECT count(*), sum(id) FROM {table}").fetchone()


def worker(n, folder, private_tmpdir, barrier, results):
    if private_tmpdir:
        os.environ["TMPDIR"] = tempfile.mkdtemp() + "/"
    src, dst = f"{folder}/src{n}.sqlite", f"{folder}/copy{n}.sqlite"
    geodiff = pygeodiff.GeoDiff()
    barrier.wait()  # start all processes at the same moment
    try:
        geodiff.make_copy("sqlite", "", src, "sqlite", "", dst)
    except Exception as exc:
        results.put("make_copy() raised: " + str(exc).strip().splitlines()[-1].split(":")[0])
        return
    same = summary(dst, f"t{n}") == summary(src, f"t{n}")
    results.put("ok" if same else "make_copy() succeeded, but the copy differs from the source")


if __name__ == "__main__":
    private_tmpdir = "--private-tmpdir" in sys.argv
    mp.set_start_method("spawn")  # separate processes, like separate workers

    folder = tempfile.mkdtemp()
    for n in range(PROCESSES):  # every process copies its own database, with its own table
        db = sqlite3.connect(f"{folder}/src{n}.sqlite")
        db.execute(f"CREATE TABLE t{n} (id INTEGER PRIMARY KEY, v TEXT)")
        db.executemany(f"INSERT INTO t{n} VALUES (?, 'x')", ((i,) for i in range(ROWS + n)))
        db.commit()
        db.close()

    outcomes = Counter()
    for _ in range(ROUNDS):
        barrier, results = mp.Barrier(PROCESSES), mp.Queue()
        procs = [
            mp.Process(target=worker, args=(n, folder, private_tmpdir, barrier, results))
            for n in range(PROCESSES)
        ]
        for p in procs:
            p.start()
        outcomes.update(results.get() for _ in procs)
        for p in procs:
            p.join()

    print(f"{ROUNDS * PROCESSES} make_copy() calls, {PROCESSES} processes at a time:")
    for outcome, count in outcomes.most_common():
        print(f"  {count:3d} x {outcome}")

Results (pygeodiff 2.3.1 from PyPI, Ubuntu 22.04, Python 3.10), 40 make_copy() calls per run:

correct raised Table does not exist succeeded, but the copy differs from the source
default (shared /tmp), 3 runs 2, 3, 4 19, 20, 24 18, 13, 17
--private-tmpdir (control), 3 runs 40, 40, 40 0 0

With 2 processes at a time only 2 of 20 calls were correct. The wrong copies I saw include a table with 0 rows and a table with extra rows that are not in the source (100003 rows with an id sum of 5000250003, from a source with 100000 rows and 4999950000).

The script starts all processes at the same instant, which is the worst case. A collision needs overlapping calls in processes that have made the same number of rand() calls so far, which is always the case for freshly started processes.

The same problem in the test suite

ctest -j fails intermittently on master, e.g. CAPITest.test_copy and some PostgresDriverTest tests, because geodiff_c_api_test and geodiff_driver_postgres_test both open /tmp/geodiff_changeset1804289383 (strace). On unmodified master, starting PostgresDriverTest.test_makeCopy 30 to 100 ms before CAPITest.test_copy makes one or both of them fail often (at a 70 ms offset, 10 of 12 Postgres runs failed). A serial ctest is not affected.

Suggested fix

Create temporary files with names that cannot collide between processes, preferably created atomically: mkstemp() on POSIX and GetTempFileNameW() on Windows, instead of deriving names from rand(). std::filesystem is probably not an option, since the macOS Intel wheels are built with MACOSX_DEPLOYMENT_TARGET=10.9. A smaller improvement would be to add the process id to the names and seed from std::random_device.

Workaround until then: use a different TMPDIR for every process (verified by the control run above).

Environment

pygeodiff 2.3.1 (PyPI wheel) and master 82fe7d8, Ubuntu 22.04.5 (Linux 5.15), Python 3.10.12. Not tested on Windows or macOS.

Note

This issue was found during AI testing for https://github.com/MerginMaps/geodiff/pull/266. I thought it was worth mentioning. Feel free to close it if it's not related.

Dominant language
Python
Stars
185
Forks
22
Avg merge
1h 31m
Merged PRs (30d)
2

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from MerginMaps/geodiff

All issues in MerginMaps/geodiff

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.