[Bug Report] Severe VRAM Allocation Instability in PyTorch after llama-cpp-python is Imported

未關閉
#2,060 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
5/5
預估耗時
一週以上
新手友好度
30/100
Issue 類型
缺陷
描述清晰度
基本清楚
活躍度
停滯
技術堆疊
python, pytorch

研究方向

該 issue 描述了 llama-cpp-python 與 PyTorch 之間的底層 CUDA context 衝突。首先檢查 llama-cpp-python 的 bindings 中的 CUDA 初始化程式碼,尤其是 import 時附近的程式碼。尋找任何 CUDA context 建立操作或記憶體配置器設定。在一個會 import llama-cpp-python、接著載入大型 PyTorch 模型的最小環境中重現問題,同時監控 VRAM 配置模式。「Done」意味著找出導致衝突的特定呼叫或設定,並提出修正方案。

由索引模型根據 Issue 內容生成。

描述

Image Image

Prerequisites

Expected Behavior

When loading a large PyTorch-based model (e.g., an SDXL checkpoint >10GB) within a host application (ComfyUI), I expect the GPU's dedicated memory (VRAM) usage to increase smoothly and stabilize at the required level for the model. This process should be fast and efficient.

Current Behavior

The VRAM allocation is extremely erratic and inefficient. The memory usage graph shows a severe sawtooth/spiky pattern—it spikes up, drops slightly, spikes again, and repeats this cycle until the model is finally loaded. This process is significantly slower than normal and suggests a fundamental conflict in memory management at the CUDA level.

This behavior only occurs when a component that imports llama-cpp-python is loaded by the host application at startup. The issue manifests before any llama-cpp-python functions are actively called; the mere act of importing the library is enough to trigger this conflict with subsequent PyTorch operations.

Environment and Context

  • Hardware:

    • GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (94.5 GB VRAM)
  • Operating System:

    • Windows 11
  • SDK and Software Versions:

    • Python Version: 3.12.11
    • Compiler: MSVC / Visual Studio 2022 used to compile for CUDA
    • CUDA Toolkit: 12.8 (locally compiled)
    • NVIDIA Driver Version: 580.97
    • Host Application: ComfyUI (Release 2025-08-27 or newer)
  • Key Python Packages:

    • llama-cpp-python: 0.3.16
    • torch: 2.8.0+cu128
    • diskcache: 5.6.3

Failure Information (for bugs)

This report details a deep-seated conflict between llama-cpp-python and torch regarding CUDA context and/or VRAM management.

Investigation Summary & Key Findings

We conducted an extensive, multi-day investigation to isolate this issue, and the results were conclusive:

  1. The problem is triggered by the import of llama-cpp-python. Any host application that loads this library at startup will cause subsequent, unrelated torch VRAM allocations to become unstable. Disabling the components that import llama-cpp-python completely resolves the issue.

  2. The diskcache dependency is a Red Herring. We initially suspected the diskcache library was the cause. To test this, we performed a definitive experiment: we hid the real diskcache library and replaced it with a completely inert, non-functional fake Python package that only contained an empty Cache class.

    • Result: The VRAM instability problem persisted even with the fake library.
    • Log Confirmation: The application log confirmed our fake library was being used:
      --- 警告:正在使用一個偽造的 (FAKE) diskcache 函式庫 ---
      Set up system temp directory: C:\Users\Win\AppData\Local\Temp\latentsync_16880d62
      
    • Conclusion: This proves that diskcache's code is not the active cause. The problem is llama-cpp-python itself. The initial correlation was due to diskcache being a mandatory dependency; uninstalling it simply prevented llama-cpp-python from being imported successfully.
Final Hypothesis

The evidence overwhelmingly suggests a low-level conflict between how llama-cpp-python (especially a version compiled against a modern CUDA 12.8 toolkit) initializes its CUDA context and how PyTorch expects to manage the VRAM. The initialization of llama-cpp-python appears to "pollute" or "fragment" the VRAM state, making subsequent large-scale allocations by PyTorch highly inefficient.

Steps to Reproduce

  1. Set up a ComfyUI environment with PyTorch and the system components listed above.
  2. Compile and install llama-cpp-python with CUDA 12.8 support.
  3. Install any ComfyUI custom node that imports llama-cpp-python at startup (e.g., ComfyUI-JoyCaption or ComfyUI-Searge-LLM-Node).
  4. Launch ComfyUI. The server will start, and llama-cpp-python will be imported into the process.
  5. Load a standard workflow designed to load a large PyTorch-based checkpoint model (e.g., an SDXL checkpoint, >10GB).
  6. Observe the "Dedicated GPU Memory" graph in Windows Task Manager. You will see the erratic, spiky allocation pattern instead of a smooth ramp-up.

Note on llama.cpp comparison: This issue is not about the performance or functionality of llama-cpp-python's inference, but rather a resource conflict it creates with another major library (torch). Therefore, comparing behavior against llama.cpp's ./main executable is not applicable, as the issue requires both runtimes to be active in the same Python process.

主要語言
Python
星號
10.7k
分支
1.5k
平均合併
1 小時 40 分鐘
30 天內合併 PR
5

環境準備

  • 沒有 Dockerfile 或 Docker Compose 檔案
  • 沒有 Pull Request 範本
  • 閱讀貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

abetlen/llama-cpp-python 的其他 Issue

查看 abetlen/llama-cpp-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。