Scores are stored in a 32-bit NumPy array even when K and V are quantized

El issue apunta a la línea 460 de llama_cpp/llama.py, donde se crea un array de NumPy con dtype=np.single. La corrección consiste en cambiar el dtype para que coincida con la cuantización de los tensores K y V (por ejemplo, np.half para 16 bits). Primero, examina el código circundante para entender cómo se usa el array scores y qué dtype tienen los tensores cuantizados. Después, modifica el dtype en consecuencia y prueba ejecutando el comando del servidor con un contexto grande para asegurarte de que no se produzca ningún error de memoria.

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

It should load the scores into an array with the appropriate data type

Current Behavior

Instead, it loads them into 32 bit ndarray

Environment and Context

Windows 10
Python 3.11.9
Latest CUDA 12.1 wheel as of now
RTX 3090
RTX 3070 (Hidden to application in this test)
32GB RAM

Failure Information

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\__main__.py", line 100, in <module>
    main()
    app = create_app(
          ^^^^^^^^^^^
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\app.py", line 150, in create_app
    set_llama_proxy(model_settings=model_settings)
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\app.py", line 70, in set_llama_proxy
    _llama_proxy = LlamaProxy(models=model_settings)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\model.py", line 31, in __init__
    self._current_model = self.load_llama_from_model_settings(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\server\model.py", line 236, in load_llama_from_model_settings
    _model = create_fn(
             ^^^^^^^^^^
  File "C:\Users\Ethan\miniconda3\envs\torch\Lib\site-packages\llama_cpp\llama.py", line 460, in __init__
    self.scores: npt.NDArray[np.single] = np.ndarray(
                                          ^^^^^^^^^^^
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 113. GiB for an array with shape (200000, 151552) and data type float32

Steps to Reproduce

Use the openai server with this command (or similar):
python -m llama_cpp.server --model glm-4-9b-chat-1m-Q4_0.gguf --flash_attn true --type_k 6 --type_v 6 --n_gpu_layers -1 --n_ctx 200000

Solution

The issue is on this line:
https://github.com/abetlen/llama-cpp-python/blob/main/llama_cpp/llama.py#L460C9-L462C10
I set it to dtype=np.half and it worked, but my kv quants are still at 6bit so I think it could go lower.