BUG: parallel deadlock in final petsc garbage cleanup

Aperta
  • Qualcosa non va?
#5,618 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, distributed-systems

Direzione di ricerca

Start with the linked minimal reproducer (mfe.py) and run it on two cores; the issue reports a hang during final cleanup after it prints “done.” Compare the two rank backtraces in PetscFinalize and PetscGarbageCleanup, including the MPIAllReduce and MPI_Comm_Dup paths. The issue does not identify a Firedrake source file or an agreed fix, so done would mean reproducing the hang and establishing a fix that lets both ranks finalize.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

Describe the bug
This is a deadlock that's been observed in the CI of Thetis (https://github.com/thetisproject/thetis/issues/459) and animate (towards the end of https://github.com/mesh-adaptation/animate/pull/250): when running the parallel tests in pytest, all tests pass cleanly, pytest prints its final report but the process seems to be hanging. This can be reproduced in running the exact same sequence of steps in the same containers locally, and attaching gdb at that point shows one of the processes being stuck in an MPIAllReduce in PetscFinalize->PetscGarbageCleanup and the other in MPI_Comm_Dup.

I've asked Claude to debug this. As I understand it, this may happen if:

  • between calls to petsc garbage cleanup (such as the one we do in NonlinearVariationalSolver.solve() ) one of the ranks has called gc.collect inbetween but the other has not
  • there's a petsc parent object P with a reference to petsc child object C, where the parent has a python wrapper that's non-cyclical but the child wrapper is part of a python ref cycle
  • child C and everything in its cycle gets deleted. On rank 0 python does a garbage collect and because C is still being referenced by P, Cs refcount is simply decreased from 2 to 1, and does not yet end up in the garbage map. On rank 1 however python does not yet garbage collect, Cs refcount stays 2
  • Now P gets deleted. Because the python wrapper is not in a cycle, P is immediately put in the garbage map on both ranks
  • Now we call petsc garbage cleanup. Because both ranks agree that P can be destroyed they both call PetscDestroy(P) for the final clean up. However that in turn call PetscDestroy(C). On rank 0, Cs refcount 1->0 and it is actually freed. On rank 1, Cs refcount 2->1
  • When rank 1 finally calls gc.collect it puts C in the garbage map, but rank 0 doesn't because Cs python wrapper is no longer there. Therefore in petscs garbage cleanup it never calls the final destroy on C
  • In PetscFinalize when it tries to destroy all objects associated with comm_world, on rank 0 it thinks there's nothing left, on rank 1 there is. It therefore goes into a separate code path

If of it interest, this is my conversation with Claude: claude_parallel_CI_hang.txt (relevant conclusion starts from line 399)

Steps to Reproduce

I've asked Claude to produce a MFE: mfe.py
When run on 2 cores this hangs in the final cleanup after printing done

Note 1: according to Claude this issue is not fixed by https://gitlab.com/petsc/petsc/-/merge_requests/9709 It hot-patched (with a LD_PRELOAD for garbage.o), but the MFE still hangs, and so does animates CI.

Note 2: the behaviour may change depending on whether slepc is installed. This is another thing Claude flagged, despite our best efforts in petsctools to only call petsc initialize once, when we import slepc4py it also always calls petsc initialize. This means that the atexit python hook gets registered twice, and thus petsc garbage collection on comm_world gets called twice. For me the MFE hangs on a system with and without slepc, but the backtrace is different.

What I don't know is what the fix is: Claude suggests we shouldn't call PETSc.garbage_cleanup() without also explicitly calling gc.collect() - as we do in every solve - but maybe petsc's garbage collect should be made more robust for this?

Lingua principale
Python
Stelle
684
Fork
204
Merge medio
4g 14h
PR unite (30g)
46

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di firedrakeproject/firedrake

Tutte le issue di firedrakeproject/firedrake

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.