Scores are stored in a 32-bit NumPy array even when K and V are quantized

此 issue 指向 llama_cpp/llama.py 第 460 行，該處使用 dtype=np.single 建立了 NumPy 陣列。修正方式是修改 dtype，使其符合 K 和 V 張量的量化方式（例如，16 位使用 np.half）。首先，檢查周圍的程式碼，以了解 scores 陣列的使用方式，以及量化張量的 dtype。接著相應地修改 dtype，並透過使用較大 context 執行伺服器命令進行測試，以確保不會發生記憶體錯誤。

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

It should load the scores into an array with the appropriate data type

Current Behavior

Instead, it loads them into 32 bit ndarray

Environment and Context

Windows 10
Python 3.11.9
Latest CUDA 12.1 wheel as of now
RTX 3090
RTX 3070 (Hidden to application in this test)
32GB RAM

Failure Information

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\__main__.py", line 100, in <module>
    main()
    app = create_app(
          ^^^^^^^^^^^
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\app.py", line 150, in create_app
    set_llama_proxy(model_settings=model_settings)
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\app.py", line 70, in set_llama_proxy
    _llama_proxy = LlamaProxy(models=model_settings)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\model.py", line 31, in __init__
    self._current_model = self.load_llama_from_model_settings(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\model.py", line 236, in load_llama_from_model_settings
    _model = create_fn(
             ^^^^^^^^^^
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\llama.py", line 460, in __init__
    self.scores: npt.NDArray[np.single] = np.ndarray(
                                          ^^^^^^^^^^^
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 113. GiB for an array with shape (200000, 151552) and data type float32

Steps to Reproduce

Use the openai server with this command (or similar):
python -m llama_cpp.server --model glm-4-9b-chat-1m-Q4_0.gguf --flash_attn true --type_k 6 --type_v 6 --n_gpu_layers -1 --n_ctx 200000

Solution

The issue is on this line:
https://github.com/abetlen/llama-cpp-python/blob/main/llama_cpp/llama.py#L460C9-L462C10
I set it to dtype=np.half and it worked, but my kv quants are still at 6bit so I think it could go lower.