simonw/llm-mlc

Error when using other MLC models

Open

#8 opened on Aug 13, 2023

View on GitHub
 (7 comments) (1 reaction) (0 assignees)Python (11 forks)github user discovery
bughelp wanted

Repository metrics

Stars
 (192 stars)
PR merge metrics
 (PR metrics pending)

Description

The examples with Llama2 in the documentation work just fine but when I tried to download and use a different MLC model I get an error. The model is shown in the list of available models so I believe it was installed correctly.

Steps to reproduce

  1. Download model: llm mlc download-model https://huggingface.co/mlc-ai/mlc-chat-WizardMath-7B-V1.0-q4f16_1
  2. Confirm model is installed: llm mlc models
  3. Attempt to use the model: llm -m mlc-chat-WizardMath-7B-V1.0-q4f16_1 "What is the square root of 5012?"

Error message

Error: Cannot find the model library that corresponds to `WizardMath-7B-V1.0-q4f16_1`.
`WizardMath-7B-V1.0-q4f16_1` is either provided in the `chat_config` you passed in, or specified in /Users/weatherh/Code/LLM-CLI/mlc/dist/prebuilt/mlc-chat-WizardMath-7B-V1.0-q4f16_1/mlc-chat-config.json.

Thank you for sharing this tool. It is by far one of the best and one of the best documented.

Contributor guide

Research direction
Inspect the code that resolves the model library path from the chat config. Check if the library file exists at the expected location and if the naming convention is consistent across models.
Tech stack
python
Domain
cli
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70