lm-sys/FastChat

Chat Arena version of fastchat-t5-3b-v1.0 provides more refined answers than standard Huggingface model.

Open

#1264 opened on May 15, 2023

View on GitHub
 (26 comments) (3 reactions) (1 assignee)Python (38,959 stars) (4,736 forks)batch import
documentationgood first issuequestion

Description

Hello - I notice that the chat arena version of fastchat-t5-3b-v1.0 provides quite different answers from the model compared to when it is downloaded manually and run using fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0. What is the reason for this difference? Are there any special prompt templates or prompt tuning that gives the chat arena version its better responses?

Contributor guide

Tech stack
pythonpytorch
Domain
machine learningai
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
needs maintainer response
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
PythonFastChat basicsPrompt engineering basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate the differences between the Chat Arena version and standard Huggingface model of fastchat t5-3b v1.0. Check the FastChat source code for any special prompt templates or prompt tuning used in the arena mode (e.g., files like fastchat/serve/ or fastchat/train/). Look at the model card and any configuration differences. Consider the discussion in the issue comments for hints.