Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug]: SIGSEGV at multi-rank iris.iris() init on mixed-architecture AMD GPUs

Open
  • Is something wrong?
#556 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 4 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
40/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend, distributed-systems

Research direction

The crash occurs in iris/host/distributed/helpers.py at line 321 in distributed_barrier, called from setup_fd_infrastructure. Start by examining the barrier implementation in torch/distributed/distributed_c10d.py line 5230. Run a minimal test with HIP_VISIBLE_DEVICES to isolate mixed-architecture effects. Check for device context creation or IPC queries before the barrier. The goal is to reproduce the segfault and understand why it only happens with heterogeneous GPUs.

Written by the indexing model from the issue text.

Description

bug iris test

Describe the bug

iris.iris(heap_size) segfaults (SIGSEGV) at multi-rank initialization, before
any kernel is launched, on a machine with three AMD GPUs of different
architectures. Single-rank works.

To Reproduce

On a box where torch.cuda.device_count() == 3 and the devices are
mixed-generation:

python tests/run_tests_distributed.py tests/unittests/ --num_ranks 2 -x

Result: one rank dies with Signal 11 (SIGSEGV), the surviving rank is torn
down with SIGTERM. --num_ranks 1 passes 84/84 in test_empty.py.

Stack trace / error message

Fatal Python error: Segmentation fault
Current thread 0x00007ff10c2e6740 (most recent call first):
  File "torch/distributed/distributed_c10d.py", line 5230 in barrier
  File "torch/distributed/c10d_logger.py", line 83 in wrapper
  File "iris/host/distributed/helpers.py", line 321 in distributed_barrier
  File "iris/host/distributed/fd_passing.py", line 236 in setup_fd_infrastructure
  File "iris/host/memory/symmetric_heap.py", line 169 in __init__
  File "iris/host/iris.py", line 115 in __init__
  File "iris/host/iris.py", line 1390 in iris

Expected behavior

iris.iris() initializes at 2+ ranks on mixed-architecture AMD GPUs, or fails
with a diagnosable error rather than a segfault.

Environment

  • OS: Arch Linux / CachyOS, kernel 7.2.6-1-cachyos
  • ROCm: 7.14 (torch reports hip: 7.14.60850)
  • torch: 2.12.0+rocm7.14
  • triton: 3.7.1
  • iris: main @ 5b46b8e
  • GPUs: RX 9070 XT (gfx1201) + Vega Frontier Edition (gfx900:xnack-) +
    integrated Radeon (gfx1036) — three generations in one box
  • torch.cuda.device_count() → 3

Additional context — why I suspect it is the mixed-arch set, not generic multi-rank

Three observations, in order of strength:

  1. A bare torch.distributed test with no iris involved passes. Two
    processes, init_process_group("gloo"), dist.barrier(),
    destroy_process_group() — all clean. So the process group itself forms and
    barriers fine at this PyTorch version. The crash is specific to iris's
    initialization path, or to a resource iris allocates during it.

  2. The crash is immediately after a plain dist.barrier() (helpers.py:321),
    inside setup_fd_infrastructure — i.e. before the dmabuf/fd mesh logic that
    the surrounding code is really about. That makes the barrier itself look
    like the trigger surface rather than a victim of later corruption.

  3. All three ranks default to device ordinal = rank, so rank 0/1/2 land on
    gfx1201, gfx900, and gfx1036 respectively. If any per-device context
    (or a peer-access / IPC capability query) is created before the barrier,
    a heterogeneous set is the obvious place for a driver-level fault — and
    homogeneous CI hardware would not surface it.

I have not been able to bisect further than this locally. If it is useful I am
happy to run a targeted probe against any specific hypothesis (e.g. forcing
same-arch device selection via HIP_VISIBLE_DEVICES, or stubbing the
setup_fd_infrastructure barrier) — just say which is most informative.

I am filing the docs/example-test fixes separately in a PR; this issue is the
blocker that stopped me from validating those end-to-end at >1 rank.

Dominant language
Python
Stars
202
Forks
52
Avg merge
6d 1h
Merged PRs (30d)
18

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ROCm/iris

All issues in ROCm/iris

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.