make_copy() in concurrent processes fails or silently returns a wrong copy (unseeded rand() gives identical temp file names)
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 40/100
Línea de trabajo
The collision starts at the temp-file name built from rand() in makeCopy() in geodiff/src/geodiff.cpp, and randomString() and the TmpFile destructor in geodiff/src/geodiffutils.cpp handle the same naming and deletion. Read those, then run the Python reproduction in the issue, which should fail on a shared /tmp and pass with --private-tmpdir. Done means concurrent make_copy() calls return correct copies on a shared TMPDIR and ctest -j passes without collisions on the changeset file.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Summary
GEODIFF_makeCopy() (make_copy() in Python) writes the source data to a temporary changeset whose name comes from rand(), and nothing in geodiff ever seeds rand(). Every process therefore generates the same sequence of "random" names, so separate processes running make_copy() at the same time share one temporary file (and the first to finish deletes it). In the reproduction below this makes make_copy() raise Table does not exist, or return success with a copy that differs from the source.
The same thing makes ctest -j fail intermittently on master (see the end).
Where
At commit 82fe7d8 (2.3.1 + 2):
geodiff.cpp#L306:TmpFile tmpFileChangeset( tmpdir( ) + "geodiff_changeset" + std::to_string( rand() ) );inmakeCopy()TmpFile::~TmpFile()removes that file by name, so one process deletes the file another one is still using.srand()andstd::random_deviceare not used anywhere in the repository.- On glibc the first
rand()of every process returns1804289383, so a freshly started process always uses/tmp/geodiff_changeset1804289383(confirmed withstrace). randomString()(geodiffutils.cpp#L655) uses the same unseededrand()and names the temporary files ofGEODIFF_createChangesetDr()(_gpkg-xxxxxx, L392, L407) and ofGEODIFF_rebaseEx()(geodiff_xxxxxx_base2theirs.binetc., L886). I only reproduced the problem withmake_copy(); these two I know only from reading the code.
Reproduction
Several independent processes call make_copy() at the same moment, each on its own database, and check that each copy matches its own source. Run with pip install pygeodiff.
"""
Independent processes call make_copy() at the same moment, each on its own database.
python repro_makecopy_race.py # default: all processes share /tmp
python repro_makecopy_race.py --private-tmpdir # control: every process has its own TMPDIR
"""
import multiprocessing as mp
import os
import sqlite3
import sys
import tempfile
from collections import Counter
import pygeodiff
ROWS, PROCESSES, ROUNDS = 100_000, 4, 10
def summary(path, table):
"""(row count, sum of ids) of a table, None if it does not exist"""
db = sqlite3.connect(path)
if not db.execute("SELECT 1 FROM sqlite_master WHERE name = ?", (table,)).fetchone():
return None
return db.execute(f"SELECT count(*), sum(id) FROM {table}").fetchone()
def worker(n, folder, private_tmpdir, barrier, results):
if private_tmpdir:
os.environ["TMPDIR"] = tempfile.mkdtemp() + "/"
src, dst = f"{folder}/src{n}.sqlite", f"{folder}/copy{n}.sqlite"
geodiff = pygeodiff.GeoDiff()
barrier.wait() # start all processes at the same moment
try:
geodiff.make_copy("sqlite", "", src, "sqlite", "", dst)
except Exception as exc:
results.put("make_copy() raised: " + str(exc).strip().splitlines()[-1].split(":")[0])
return
same = summary(dst, f"t{n}") == summary(src, f"t{n}")
results.put("ok" if same else "make_copy() succeeded, but the copy differs from the source")
if __name__ == "__main__":
private_tmpdir = "--private-tmpdir" in sys.argv
mp.set_start_method("spawn") # separate processes, like separate workers
folder = tempfile.mkdtemp()
for n in range(PROCESSES): # every process copies its own database, with its own table
db = sqlite3.connect(f"{folder}/src{n}.sqlite")
db.execute(f"CREATE TABLE t{n} (id INTEGER PRIMARY KEY, v TEXT)")
db.executemany(f"INSERT INTO t{n} VALUES (?, 'x')", ((i,) for i in range(ROWS + n)))
db.commit()
db.close()
outcomes = Counter()
for _ in range(ROUNDS):
barrier, results = mp.Barrier(PROCESSES), mp.Queue()
procs = [
mp.Process(target=worker, args=(n, folder, private_tmpdir, barrier, results))
for n in range(PROCESSES)
]
for p in procs:
p.start()
outcomes.update(results.get() for _ in procs)
for p in procs:
p.join()
print(f"{ROUNDS * PROCESSES} make_copy() calls, {PROCESSES} processes at a time:")
for outcome, count in outcomes.most_common():
print(f" {count:3d} x {outcome}")
Results (pygeodiff 2.3.1 from PyPI, Ubuntu 22.04, Python 3.10), 40 make_copy() calls per run:
| correct | raised Table does not exist |
succeeded, but the copy differs from the source | |
|---|---|---|---|
default (shared /tmp), 3 runs |
2, 3, 4 | 19, 20, 24 | 18, 13, 17 |
--private-tmpdir (control), 3 runs |
40, 40, 40 | 0 | 0 |
With 2 processes at a time only 2 of 20 calls were correct. The wrong copies I saw include a table with 0 rows and a table with extra rows that are not in the source (100003 rows with an id sum of 5000250003, from a source with 100000 rows and 4999950000).
The script starts all processes at the same instant, which is the worst case. A collision needs overlapping calls in processes that have made the same number of rand() calls so far, which is always the case for freshly started processes.
The same problem in the test suite
ctest -j fails intermittently on master, e.g. CAPITest.test_copy and some PostgresDriverTest tests, because geodiff_c_api_test and geodiff_driver_postgres_test both open /tmp/geodiff_changeset1804289383 (strace). On unmodified master, starting PostgresDriverTest.test_makeCopy 30 to 100 ms before CAPITest.test_copy makes one or both of them fail often (at a 70 ms offset, 10 of 12 Postgres runs failed). A serial ctest is not affected.
Suggested fix
Create temporary files with names that cannot collide between processes, preferably created atomically: mkstemp() on POSIX and GetTempFileNameW() on Windows, instead of deriving names from rand(). std::filesystem is probably not an option, since the macOS Intel wheels are built with MACOSX_DEPLOYMENT_TARGET=10.9. A smaller improvement would be to add the process id to the names and seed from std::random_device.
Workaround until then: use a different TMPDIR for every process (verified by the control run above).
Environment
pygeodiff 2.3.1 (PyPI wheel) and master 82fe7d8, Ubuntu 22.04.5 (Linux 5.15), Python 3.10.12. Not tested on Windows or macOS.
Note
This issue was found during AI testing for https://github.com/MerginMaps/geodiff/pull/266. I thought it was worth mentioning. Feel free to close it if it's not related.
- Lenguaje dominante
- Python
- Estrellas
- 185
- Forks
- 22
- Merge medio
- 1 h 31 min
- PR fusionados (30 d)
- 2
Preparar el entorno
- Sin Dockerfile ni archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de MerginMaps/geodiff
-
GEODIFF_changesCount aborts the process on a truncated changeset (uncaught GeoDiffException)Posiblemente ocupada @mohanedmoh la tomó hace 21 días. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
MerginMaps/geodiff#272 · 1 comentario ·
-
Dificultad 4/5 3-5 días Aptitud para principiantes 25/100
MerginMaps/geodiff#275 ·
-
enhancement
Dificultad 4/5 3-5 días Aptitud para principiantes 46/100
MerginMaps/geodiff#274 · 1 reacción ·
-
Dificultad 4/5 3-5 días Aptitud para principiantes 48/100
MerginMaps/geodiff#262 ·
-
bug data corruption - sync failure - crash
Dificultad 4/5 3-5 días Aptitud para principiantes 48/100
MerginMaps/geodiff#255 ·
Todos los issues de MerginMaps/geodiff
Issues similares
-
bug(mcp): handle malformed and partial reranker responses safelyPosiblemente ocupada @KamilEjaz890 la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 85/100
kubeflow/docs-agent#301 ·
Los mantenedores suelen responder en 4 días
-
Update the demo docsAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 72/100
pulp/pulp_workflow#78 ·
-
Invalid syntax in sshtunnelAbiertoBug
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
pgadmin-org/pgadmin4#10797 ·
Los mantenedores suelen responder en 2 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100
gcivil-nyu-org/team5-mon-fall26#132 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 1/5 Menos de una hora Aptitud para principiantes 76/100
gcivil-nyu-org/team4-mon-fall26#291 ·
Los mantenedores suelen responder en 1 día