[molab] [gpu] [driver] RTX PRO 6000 (SM120) sessions on NVIDIA driver 595.71.05 have ~33% lower L2 bandwidth than on 580.178.04

Open
#11,453 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python, pytorch

Research direction

The report provides a PyTorch CUDA benchmark and nvidia-smi commands, but names no marimo files or tests to change. Start by checking whether GPU driver versions are controlled by marimo or by the GPU session provider; compare the reported 580 and 595 host measurements. Done means identifying the responsible layer and addressing the request to investigate the performance difference or provide a way to select a 580 host.

Written by the indexing model from the issue text.

Description

bug
Describe the bug

molab GPU sessions (RTX PRO 6000 Blackwell Server Edition) land on hosts with two different NVIDIA drivers, and the 595 hosts are consistently slower for the same notebook and code.

Both hosts have the same GPU, the same VBIOS (98.02.74.00.01) and ECC enabled. Both stay at P0 with full SM and memory clocks (2,430 / 12,481 MHz) and no clock-event reasons. The difference is in the L2/memory path:

driver 595.71.05 (slow) driver 580.178.04 (fast) fast/slow
tensor cores, register-only FP8 MMA (TFLOPS) 932 931 1.00
cuBLAS bf16 / fp8 GEMM 8192³ (TFLOPS) 404 / 789 408 / 785 1.00
L2-resident copy 16 MiB / read 64 MiB (GB/s) 3,086 / 2,740 4,186 / 3,625 1.36 / 1.32
DRAM copy 1 GiB / read 2 GiB (GB/s) 1,358 / 1,319 1,460 / 1,526 1.08 / 1.16
FP8 rowwise GEMM 61440×7680→1408 (TFLOPS) 615 722 1.17

Things I checked on the slow host:

  • The effective SM clock, read in-kernel via %clock64/%globaltimer, is 2,429 MHz, including inside the GEMMs.
  • There is no power capping: the counter doesn't move.
  • The GPU is not stuck at P1; the CudaNoStablePerfLimit profile has no effect.
  • L2 capacity is the full 128 MiB.

So it looks like a lower L2/crossbar clock or config on the 595 hosts, possibly from the GSP firmware that ships with that driver. Possibly it's a per-host issue instead.

Request: could the GPU nodes be pinned to the 580 driver, or could you check with your GPU provider why the 595 nodes have lower L2 bandwidth? A way to choose or guarantee a 580 node would also work.

Workaround: check nvidia-smi --query-gpu=driver_version --format=csv,noheader at session start, and restart the session until it reports 580.

GPU: RTX PRO 6000 Blackwell Server Edition, torch 2.14.1+cu132, triton 3.8.0. Slow host: driver 595.71.05 / CUDA 13.2. Fast host: driver 580.178.04 / CUDA 13.0.

Will you submit a PR?
  • Yes
Environment
Environment
{
  "marimo": "0.25.2",
  "editable": false,
  "location": "/usr/local/lib/python3.13/site-packages/marimo",
  "OS": "Linux",
  "OS Version": "4.19.0-gvisor",
  "Processor": "",
  "Python Version": "3.13.11",
  "Locale": "C",
  "Binaries": {
    "Browser": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
    "Node": "v22.23.4",
    "uv": "0.12.1 (x86_64-unknown-linux-gnu)"
  },
  "Dependencies": {
    "click": "8.5.0",
    "docutils": "0.23",
    "itsdangerous": "2.2.0",
    "jedi": "0.19.2",
    "markdown": "3.10.2",
    "narwhals": "2.24.0",
    "packaging": "26.3",
    "psutil": "7.2.2",
    "pygments": "2.21.0",
    "pymdown-extensions": "10.21.3",
    "pyyaml": "6.0.3",
    "starlette": "1.3.1",
    "tomlkit": "0.15.1",
    "typing-extensions": "4.16.0",
    "uvicorn": "0.51.0",
    "websockets": "16.1.1"
  },
  "Optional Dependencies": {
    "altair": "6.2.2",
    "anywidget": "0.11.0",
    "duckdb": "1.5.4",
    "loro": "1.13.1",
    "nbformat": "5.10.4",
    "openai": "3.19.1",
    "pandas": "3.1.0",
    "polars": "1.42.1",
    "pyarrow": "26.0.0",
    "pytest": "9.1.1",
    "python-lsp-ruff": "2.3.1",
    "python-lsp-server": "1.14.0",
    "ruff": "0.15.22",
    "sqlglot": "30.12.0",
    "ty": "0.0.61",
    "vegafusion": "2.0.3"
  },
  "Experimental Flags": {}
}
Code to reproduce

Repro (~10 s, needs torch with CUDA):

import subprocess, time, torch
print(subprocess.run(["nvidia-smi", "--query-gpu=driver_version,vbios_version,pstate,clocks.sm,clocks.mem",
                      "--format=csv,noheader"], capture_output=True, text=True).stdout.strip())
def bench(fn, s=1.0):
    fn(); torch.cuda.synchronize(); n, t0 = 0, time.perf_counter()
    while time.perf_counter() - t0 < s:
        for _ in range(5): fn()
        torch.cuda.synchronize(); n += 5
    return (time.perf_counter() - t0) / n
src = torch.empty(8 << 20, device="cuda", dtype=torch.bfloat16); dst = torch.empty_like(src)   # 16 MiB, L2-resident
print(f"L2 copy  {2 * src.nbytes / bench(lambda: dst.copy_(src)) / 1e9:.0f} GB/s  (580: ~4,190, 595: ~3,090)")
src = torch.empty(512 << 20, device="cuda", dtype=torch.bfloat16); dst = torch.empty_like(src)  # 1 GiB
print(f"DRAM copy {2 * src.nbytes / bench(lambda: dst.copy_(src)) / 1e9:.0f} GB/s  (580: ~1,460, 595: ~1,360)")
del src, dst
m, k, n = 61440, 7680, 1408
a = torch.randn(m, k, device="cuda").to(torch.float8_e5m2); b = torch.randn(n, k, device="cuda").to(torch.float8_e4m3fn).t()
sa, sb = torch.rand(m, 1, device="cuda") + 0.5, torch.rand(1, n, device="cuda") + 0.5
t = bench(lambda: torch._scaled_mm(a, b, sa, sb, out_dtype=torch.bfloat16))
print(f"fp8 GEMM {2 * m * k * n / t / 1e12:.0f} TFLOPS  (580: ~720, 595: ~615)")
Dominant language
Python
Stars
22.9k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
380

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from marimo-team/marimo

All issues in marimo-team/marimo

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.