mozilla-ai/llamafile

LLaVA loads with errors, GUI opens but doesn't produce any output

Open

#104 opened on Dec 15, 2023

View on GitHub
 (34 comments) (0 reactions) (0 assignees)C++ (24,439 stars) (1,358 forks)batch import
help wanted

Description

I renamed the file to llava-v1.5-7b-q4-server.llamafile.exe, ran it in the cmd.exe terminal. Everything looked okay (who knows? I don't know what all the output means) but it ended with this:

llama.cpp/ggml.c:9064: assert(!isnan(x)) failed (cosmoaddr2line /C/Users/shady/Desktop/llava-v1.5-7b-q4-server.llamafile.exe 4c5399 4e3822 4e7219 5c952c 5e75c3)
llama.cpp/ggml.c:9064: assert(!isnan(x)) failed (cosmoaddr2line /C/Users/shady/Desktop/llava-v1.5-7b-q4-server.llamafile.exe 4c5399 4e3822 4e7219 5c952c 5e75c3)

error: Uncaught SIGABRT (SI_TKILL) at 0 on DESKTOP-MM6QUGS pid 7460 tid 11132
  llava-v1.5-7b-q4-server.llamafile
  No error information
  Windows Cosmopolitan 3.1.2 MODE=x86_64 DESKTOP-MM6QUGS 10.0-19045

RAX 0000100080b7f950 RBX 000000000061b6f3 RDI 0000100080b7f880
RCX 000010008019d7b0 RDX 0000000000000000 RSI 00000000fffffffa
RBP 0000100080b7fbd0 RSP 0000100080b7f760 RIP 0000000000410672
 R8 0000000000000000  R9 0000000000000001 R10 0000000000000000
R11 0000000000000246 R12 0000000000000006 R13 000000000061b4bf
R14 0000100080b7fc20 R15 0000000000002aff
TLS 000010008019d760

XMM0  00000000000000000000000000000000 XMM8  00000000000000000000000000000000
XMM1  00000000000000000000000000000000 XMM9  00000000000000000000000000000000
XMM2  00000000000000000000000000000000 XMM10 00000000000000000000000000000000
XMM3  00000000000000000000000000000000 XMM11 00000000000000000000000000000000
XMM4  00000000000000000000000000000000 XMM12 00000000000000000000000000000000
XMM5  00000000000000000000000000000000 XMM13 00000000000000000000000000000000
XMM6  0000000000000000000000003727c5ac XMM14 00000000000000000000000000000000
XMM7  00000000000000000000000000000000 XMM15 00000000000000000000000000000000

The GUI opened up in my browser (Firefox) but I couldn't get it to produce any output. It would just freeze. I could reset the prompt and start over, but it never produces output. Running Windows 10 with an RTX 2060 graphics card

Contributor guide

Tech stack
cpp
Domain
backendclimachine learning
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
C++GPU programmingllamafile build system
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The crash occurs in llama.cpp/ggml.c line 9064 due to a NaN assertion. This is likely a floating point issue when running on GPU (RTX 2060). Investigate whether the model quantization or GPU kernel is causing NaN values. Review recent changes to ggml or CUDA backend. Check if the issue is reproducible with other models or system configurations. Also, look at the thread for any maintainer suggestions or workarounds.