Model Hallucination Occurs again and again.

Reproduziere den Bericht mit setup_env.py und run_inference.py unter Verwendung von models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf und dem bereitgestellten Prompt. Beginne damit, die vollständig generierte Antwort und die Umgebungsdetails zu erfassen; das Issue definiert derzeit nicht die erwartete Ausgabe und zeigt die angebliche Halluzination nicht. Erledigt ist die Aufgabe, wenn der Fehler reproduzierbar ist und seine Ursache und Korrektur eindeutig identifiziert sind.

python -m venv venv
╰─❮ pip install -r requirements.txt
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
python-xyz', where xyz is the package you are trying to
install.

If you wish to install a non-Arch-packaged Python package,
create a virtual environment using 'python -m venv path/to/venv'.
Then use path/to/venv/bin/python and path/to/venv/bin/pip.

If you wish to install a non-Arch packaged Python application,
it may be easiest to use 'pipx install xyz', which will manage a
virtual environment for you. Make sure you have python-pipx
installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
╰─❮ source venv/bin/activate.fish
╰─❮ pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu, https://download.pytorch.org/whl/nightly, https://download.pytorch.org/whl/cpu, https://download.pytorch.org/whl/nightly, https://download.pytorch.org/whl/cpu, https://download.pytorch.org/whl/nightly
Ignoring torch: markers 'platform_machine == "s390x"' don't match your environment
Ignoring torch: markers 'platform_machine == "s390x"' don't match your environment
Collecting numpy~=1.26.4 (from -r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 1))
Using cached numpy-1.26.4-cp314-cp314-linux_x86_64.whl
Collecting sentencepiece<0.3.0,>=0.1.98 (from -r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 2))
Using cached sentencepiece-0.2.2-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (33 kB)
Collecting transformers==4.57.6 (from -r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached transformers-4.57.6-py3-none-any.whl.metadata (43 kB)
Collecting gguf>=0.1.0 (from -r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 6))
Using cached gguf-0.19.0-py3-none-any.whl.metadata (4.1 kB)
Collecting protobuf<5.0.0,>=4.21.0 (from -r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 7))
Using cached protobuf-4.25.9-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting torch==2.11.0 (from -r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached torch-2.11.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl.metadata (29 kB)
Collecting filelock (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached filelock-3.32.2-py3-none-any.whl.metadata (2.0 kB)
Collecting huggingface-hub<1.0,>=0.34.0 (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached huggingface_hub-0.36.2-py3-none-any.whl.metadata (15 kB)
Collecting packaging>=20.0 (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Downloading packaging-26.3-py3-none-any.whl.metadata (3.5 kB)
Collecting pyyaml>=5.1 (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB)
Collecting regex!=2019.12.17 (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached regex-2026.7.19-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
Collecting requests (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached requests-2.34.2-py3-none-any.whl.metadata (4.8 kB)
Collecting tokenizers<=0.23.0,>=0.22.0 (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB)
Collecting safetensors>=0.4.3 (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)
Collecting tqdm>=4.27 (from transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached tqdm-4.70.0-py3-none-any.whl.metadata (57 kB)
Collecting typing-extensions>=4.10.0 (from torch==2.11.0->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached typing_extensions-4.16.0-py3-none-any.whl.metadata (3.3 kB)
Collecting setuptools<82 (from torch==2.11.0->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached setuptools-81.0.0-py3-none-any.whl.metadata (6.6 kB)
Collecting sympy>=1.13.3 (from torch==2.11.0->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached sympy-1.14.0-py3-none-any.whl.metadata (12 kB)
Collecting networkx>=2.5.1 (from torch==2.11.0->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached networkx-3.6.1-py3-none-any.whl.metadata (6.8 kB)
Collecting jinja2 (from torch==2.11.0->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec>=0.8.5 (from torch==2.11.0->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached fsspec-2026.7.0-py3-none-any.whl.metadata (10 kB)
Collecting hf-xet<2.0.0,>=1.1.3 (from huggingface-hub<1.0,>=0.34.0->transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Downloading hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (4.9 kB)
Collecting charset_normalizer<4,>=2 (from requests->transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached charset_normalizer-3.4.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (41 kB)
Collecting idna<4,>=2.5 (from requests->transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached idna-3.18-py3-none-any.whl.metadata (6.1 kB)
Collecting urllib3<3,>=1.26 (from requests->transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached urllib3-2.7.0-py3-none-any.whl.metadata (6.9 kB)
Collecting certifi>=2023.5.7 (from requests->transformers==4.57.6->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_legacy_llama.txt (line 4))
Using cached certifi-2026.7.22-py3-none-any.whl.metadata (2.5 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch==2.11.0->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch==2.11.0->-r /home/luca/bitnet/BitNet/3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt (line 5))
Using cached markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB)
Using cached transformers-4.57.6-py3-none-any.whl (12.0 MB)
Using cached torch-2.11.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl (190.3 MB)
Using cached sentencepiece-0.2.2-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.4 MB)
Using cached protobuf-4.25.9-cp37-abi3-manylinux2014_x86_64.whl (295 kB)
Using cached huggingface_hub-0.36.2-py3-none-any.whl (566 kB)
Downloading hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 4.3 MB/s 0:00:01
Using cached setuptools-81.0.0-py3-none-any.whl (1.1 MB)
Using cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
Using cached gguf-0.19.0-py3-none-any.whl (118 kB)
Using cached fsspec-2026.7.0-py3-none-any.whl (206 kB)
Using cached networkx-3.6.1-py3-none-any.whl (2.1 MB)
Downloading packaging-26.3-py3-none-any.whl (129 kB)
Using cached pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (794 kB)
Using cached regex-2026.7.19-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (801 kB)
Using cached requests-2.34.2-py3-none-any.whl (73 kB)
Using cached charset_normalizer-3.4.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (223 kB)
Using cached idna-3.18-py3-none-any.whl (65 kB)
Using cached urllib3-2.7.0-py3-none-any.whl (131 kB)
Using cached certifi-2026.7.22-py3-none-any.whl (136 kB)
Using cached safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (516 kB)
Using cached sympy-1.14.0-py3-none-any.whl (6.3 MB)
Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Using cached tqdm-4.70.0-py3-none-any.whl (80 kB)
Using cached typing_extensions-4.16.0-py3-none-any.whl (45 kB)
Using cached filelock-3.32.2-py3-none-any.whl (98 kB)
Using cached jinja2-3.1.6-py3-none-any.whl (134 kB)
Using cached markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (23 kB)
Installing collected packages: mpmath, urllib3, typing-extensions, tqdm, sympy, setuptools, sentencepiece, safetensors, regex, pyyaml, protobuf, packaging, numpy, networkx, MarkupSafe, idna, hf-xet, fsspec, filelock, charset_normalizer, certifi, requests, jinja2, torch, huggingface-hub, gguf, tokenizers, transformers
Successfully installed MarkupSafe-3.0.3 certifi-2026.7.22 charset_normalizer-3.4.9 filelock-3.32.2 fsspec-2026.7.0 gguf-0.19.0 hf-xet-1.6.0 huggingface-hub-0.36.2 idna-3.18 jinja2-3.1.6 mpmath-1.3.0 networkx-3.6.1 numpy-1.26.4 packaging-26.3 protobuf-4.25.9 pyyaml-6.0.3 regex-2026.7.19 requests-2.34.2 safetensors-0.8.0 sentencepiece-0.2.2 setuptools-81.0.0 sympy-1.14.0 tokenizers-0.22.2 torch-2.11.0+cpu tqdm-4.70.0 transformers-4.57.6 typing-extensions-4.16.0 urllib3-2.7.0

[notice] A new release of pip is available: 26.1.2 -> 26.2.1
[notice] To update, run: pip install --upgrade pip
╰─❮ python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
INFO:root:Compiling the code using CMake.
INFO:root:Loading model from directory models/BitNet-b1.58-2B-4T.
INFO:root:GGUF model already exists at models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
╰─❮ python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "how are you, are you good?"
warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions

Loading model...

▄▄ ▄▄
██ ██
██ ██ ▀▀█▄ ███▄███▄ ▀▀█▄ ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██ ██ ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
██ ██
▀▀ ▀▀

build : b9918-390c30775
model : models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
ftype : Q1_0
modalities : text

available commands:
/exit or Ctrl+C stop or exit
/regen regenerate the last response
/clear clear the chat history
/read add a text file
/glob add text files using globbing pattern

how are you, are you good?

Yes, I am good. I am a good person. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes, I am good. Yes,

[ Prompt: 44.9 t/s | Generation: 11.7 t/s ]