[molab] [gpu] [driver] RTX PRO 6000 (SM120) sessions on NVIDIA driver 595.71.05 have ~33% lower L2 bandwidth than on 580.178.04
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- cloud, infrastructure
Research direction
The report provides a PyTorch CUDA benchmark and nvidia-smi commands, but names no marimo files or tests to change. Start by checking whether GPU driver versions are controlled by marimo or by the GPU session provider; compare the reported 580 and 595 host measurements. Done means identifying the responsible layer and addressing the request to investigate the performance difference or provide a way to select a 580 host.
Written by the indexing model from the issue text.
Description
Describe the bug
molab GPU sessions (RTX PRO 6000 Blackwell Server Edition) land on hosts with two different NVIDIA drivers, and the 595 hosts are consistently slower for the same notebook and code.
Both hosts have the same GPU, the same VBIOS (98.02.74.00.01) and ECC enabled. Both stay at P0 with full SM and memory clocks (2,430 / 12,481 MHz) and no clock-event reasons. The difference is in the L2/memory path:
| driver 595.71.05 (slow) | driver 580.178.04 (fast) | fast/slow | |
|---|---|---|---|
| tensor cores, register-only FP8 MMA (TFLOPS) | 932 | 931 | 1.00 |
| cuBLAS bf16 / fp8 GEMM 8192³ (TFLOPS) | 404 / 789 | 408 / 785 | 1.00 |
| L2-resident copy 16 MiB / read 64 MiB (GB/s) | 3,086 / 2,740 | 4,186 / 3,625 | 1.36 / 1.32 |
| DRAM copy 1 GiB / read 2 GiB (GB/s) | 1,358 / 1,319 | 1,460 / 1,526 | 1.08 / 1.16 |
| FP8 rowwise GEMM 61440×7680→1408 (TFLOPS) | 615 | 722 | 1.17 |
Things I checked on the slow host:
- The effective SM clock, read in-kernel via
%clock64/%globaltimer, is 2,429 MHz, including inside the GEMMs. - There is no power capping: the counter doesn't move.
- The GPU is not stuck at P1; the
CudaNoStablePerfLimitprofile has no effect. - L2 capacity is the full 128 MiB.
So it looks like a lower L2/crossbar clock or config on the 595 hosts, possibly from the GSP firmware that ships with that driver. Possibly it's a per-host issue instead.
Request: could the GPU nodes be pinned to the 580 driver, or could you check with your GPU provider why the 595 nodes have lower L2 bandwidth? A way to choose or guarantee a 580 node would also work.
Workaround: check nvidia-smi --query-gpu=driver_version --format=csv,noheader at session start, and restart the session until it reports 580.
GPU: RTX PRO 6000 Blackwell Server Edition, torch 2.14.1+cu132, triton 3.8.0. Slow host: driver 595.71.05 / CUDA 13.2. Fast host: driver 580.178.04 / CUDA 13.0.
Will you submit a PR?
- Yes
Environment
Environment
{
"marimo": "0.25.2",
"editable": false,
"location": "/usr/local/lib/python3.13/site-packages/marimo",
"OS": "Linux",
"OS Version": "4.19.0-gvisor",
"Processor": "",
"Python Version": "3.13.11",
"Locale": "C",
"Binaries": {
"Browser": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
"Node": "v22.23.4",
"uv": "0.12.1 (x86_64-unknown-linux-gnu)"
},
"Dependencies": {
"click": "8.5.0",
"docutils": "0.23",
"itsdangerous": "2.2.0",
"jedi": "0.19.2",
"markdown": "3.10.2",
"narwhals": "2.24.0",
"packaging": "26.3",
"psutil": "7.2.2",
"pygments": "2.21.0",
"pymdown-extensions": "10.21.3",
"pyyaml": "6.0.3",
"starlette": "1.3.1",
"tomlkit": "0.15.1",
"typing-extensions": "4.16.0",
"uvicorn": "0.51.0",
"websockets": "16.1.1"
},
"Optional Dependencies": {
"altair": "6.2.2",
"anywidget": "0.11.0",
"duckdb": "1.5.4",
"loro": "1.13.1",
"nbformat": "5.10.4",
"openai": "3.19.1",
"pandas": "3.1.0",
"polars": "1.42.1",
"pyarrow": "26.0.0",
"pytest": "9.1.1",
"python-lsp-ruff": "2.3.1",
"python-lsp-server": "1.14.0",
"ruff": "0.15.22",
"sqlglot": "30.12.0",
"ty": "0.0.61",
"vegafusion": "2.0.3"
},
"Experimental Flags": {}
}
Code to reproduce
Repro (~10 s, needs torch with CUDA):
import subprocess, time, torch
print(subprocess.run(["nvidia-smi", "--query-gpu=driver_version,vbios_version,pstate,clocks.sm,clocks.mem",
"--format=csv,noheader"], capture_output=True, text=True).stdout.strip())
def bench(fn, s=1.0):
fn(); torch.cuda.synchronize(); n, t0 = 0, time.perf_counter()
while time.perf_counter() - t0 < s:
for _ in range(5): fn()
torch.cuda.synchronize(); n += 5
return (time.perf_counter() - t0) / n
src = torch.empty(8 << 20, device="cuda", dtype=torch.bfloat16); dst = torch.empty_like(src) # 16 MiB, L2-resident
print(f"L2 copy {2 * src.nbytes / bench(lambda: dst.copy_(src)) / 1e9:.0f} GB/s (580: ~4,190, 595: ~3,090)")
src = torch.empty(512 << 20, device="cuda", dtype=torch.bfloat16); dst = torch.empty_like(src) # 1 GiB
print(f"DRAM copy {2 * src.nbytes / bench(lambda: dst.copy_(src)) / 1e9:.0f} GB/s (580: ~1,460, 595: ~1,360)")
del src, dst
m, k, n = 61440, 7680, 1408
a = torch.randn(m, k, device="cuda").to(torch.float8_e5m2); b = torch.randn(n, k, device="cuda").to(torch.float8_e4m3fn).t()
sa, sb = torch.rand(m, 1, device="cuda") + 0.5, torch.rand(1, n, device="cuda") + 0.5
t = bench(lambda: torch._scaled_mm(a, b, sa, sb, out_dtype=torch.bfloat16))
print(f"fp8 GEMM {2 * m * k * n / t / 1e12:.0f} TFLOPS (580: ~720, 595: ~615)")
- Dominant language
- Python
- Stars
- 22.9k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 380
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from marimo-team/marimo
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
marimo-team/marimo#12027 ·
Maintainers usually reply within 1 day
-
"Clamp all outputs" leaves cells without output marked `expand_output: true`Possibly taken @marthacryan claimed this 12 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
marimo-team/marimo#11768 · 1 assignee ·
Maintainers usually reply within 1 day
-
Cell referencing a variable from a "cancelled" cell runs and raises NameError instead of being blockedMay be free again @dmadisetti claimed this 33 days ago, and no pull request is open. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
marimo-team/marimo#11307 · 1 assignee ·
Maintainers usually reply within 1 day
-
molab
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
marimo-team/marimo#11298 · 1 comment ·
Maintainers usually reply within 1 day
-
Theme change does not affect terminal colorsMay be free again @CloudCorpRecords claimed this 36 days ago, and no pull request is open. Openneeds discussion
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
marimo-team/marimo#11244 · 4 comments ·
Maintainers usually reply within 1 day
All issues in marimo-team/marimo
Similar issues
-
run_gate loads a custom tasks file from outside the working directoryPossibly taken @Ru0k3 claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
MakazhanAlpamys/Soup#3189 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
nautechsystems/nautilus_trader#5741 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/datasets#9128 ·
Maintainers usually reply within 1 day