BUG: parallel deadlock in final petsc garbage cleanup
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 35/100
- issue の種類
- バグ
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 技術スタック
- python
調査の方向性
Start with the linked minimal reproducer (mfe.py) and run it on two cores; the issue reports a hang during final cleanup after it prints “done.” Compare the two rank backtraces in PetscFinalize and PetscGarbageCleanup, including the MPIAllReduce and MPI_Comm_Dup paths. The issue does not identify a Firedrake source file or an agreed fix, so done would mean reproducing the hang and establishing a fix that lets both ranks finalize.
索引モデルが issue の本文から書いたものです。
説明
Describe the bug
This is a deadlock that's been observed in the CI of Thetis (https://github.com/thetisproject/thetis/issues/459) and animate (towards the end of https://github.com/mesh-adaptation/animate/pull/250): when running the parallel tests in pytest, all tests pass cleanly, pytest prints its final report but the process seems to be hanging. This can be reproduced in running the exact same sequence of steps in the same containers locally, and attaching gdb at that point shows one of the processes being stuck in an MPIAllReduce in PetscFinalize->PetscGarbageCleanup and the other in MPI_Comm_Dup.
I've asked Claude to debug this. As I understand it, this may happen if:
- between calls to petsc garbage cleanup (such as the one we do in NonlinearVariationalSolver.solve() ) one of the ranks has called gc.collect inbetween but the other has not
- there's a petsc parent object P with a reference to petsc child object C, where the parent has a python wrapper that's non-cyclical but the child wrapper is part of a python ref cycle
- child C and everything in its cycle gets deleted. On rank 0 python does a garbage collect and because C is still being referenced by P, Cs refcount is simply decreased from 2 to 1, and does not yet end up in the garbage map. On rank 1 however python does not yet garbage collect, Cs refcount stays 2
- Now P gets deleted. Because the python wrapper is not in a cycle, P is immediately put in the garbage map on both ranks
- Now we call petsc garbage cleanup. Because both ranks agree that P can be destroyed they both call PetscDestroy(P) for the final clean up. However that in turn call PetscDestroy(C). On rank 0, Cs refcount 1->0 and it is actually freed. On rank 1, Cs refcount 2->1
- When rank 1 finally calls gc.collect it puts C in the garbage map, but rank 0 doesn't because Cs python wrapper is no longer there. Therefore in petscs garbage cleanup it never calls the final destroy on C
- In PetscFinalize when it tries to destroy all objects associated with comm_world, on rank 0 it thinks there's nothing left, on rank 1 there is. It therefore goes into a separate code path
If of it interest, this is my conversation with Claude: claude_parallel_CI_hang.txt (relevant conclusion starts from line 399)
Steps to Reproduce
I've asked Claude to produce a MFE: mfe.py
When run on 2 cores this hangs in the final cleanup after printing done
Note 1: according to Claude this issue is not fixed by https://gitlab.com/petsc/petsc/-/merge_requests/9709 It hot-patched (with a LD_PRELOAD for garbage.o), but the MFE still hangs, and so does animates CI.
Note 2: the behaviour may change depending on whether slepc is installed. This is another thing Claude flagged, despite our best efforts in petsctools to only call petsc initialize once, when we import slepc4py it also always calls petsc initialize. This means that the atexit python hook gets registered twice, and thus petsc garbage collection on comm_world gets called twice. For me the MFE hangs on a system with and without slepc, but the backtrace is different.
What I don't know is what the fix is: Claude suggests we shouldn't call PETSc.garbage_cleanup() without also explicitly calling gc.collect() - as we do in every solve - but maybe petsc's garbage collect should be made more robust for this?
- 主要言語
- Python
- スター
- 684
- フォーク
- 204
- 平均マージ
- 4日 14時間
- マージ済み PR(30日)
- 46
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドなし
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
firedrakeproject/firedrake のほかの issue
-
BUG: NonlinearVariationalSolver errors with adj_kwargs対応中かも @dham が 151 日前に担当しました。 オープンbug
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
firedrakeproject/firedrake#5186 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
firedrakeproject/firedrake#5091 ·
メンテナーはふだん 1 日以内に返信
-
documentation
難易度 1/5 1〜3時間 初心者へのやさしさ 82/100
firedrakeproject/firedrake#3556 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 4/5 3〜5日 初心者へのやさしさ 22/100
firedrakeproject/firedrake#5644 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
Function.assign on a subset fails with a Constant, a Real Function or abs in the expression対応中かも @connorjward が 2 日前に担当しました。 オープン
firedrakeproject/firedrake#5643 · 担当者 1 名 ·
メンテナーはふだん 1 日以内に返信
firedrakeproject/firedrake の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
alltheplaces/alltheplaces#20570 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
メンテナーはふだん 1 日以内に返信
-
area:ci bug
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
catincloud-labs/dagster-dex#206 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
number_to_words raises ValueError with threshold= for ordinals ("12th") and comma-grouped strings ("1,000")対応中かも @hemtilva が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
メンテナーはふだん 1 日以内に返信
-
providers security
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
agentbreeder/agentbreeder#1031 ·
メンテナーはふだん 1 日以内に返信