inference truncate causes “output_ids” to be incorrectly sliced
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 65/100
Research direction
Read fastchat/serve/inference.py around lines 84, 89, 91, and 178, then trace how truncation changes input_echo_len relative to output_ids. Reproduce the provided max_new_tokens example and verify that the generated output no longer includes prompt fragments; also determine what the -8 adjustment represents.
Written by the indexing model from the issue text.
Description
in inference.py
- convert
input_idsto list, assign tooutput_ids
https://github.com/lm-sys/FastChat/blob/34c585ced4c0b30d2264c519f626b4c5cb924216/fastchat/serve/inference.py#L84 - truncate
input_ids, and get length assign toinput_echo_len, butoutput_idsnot truncate,input_echo_lenlength is not equal tolen(output_ids)
https://github.com/lm-sys/FastChat/blob/34c585ced4c0b30d2264c519f626b4c5cb924216/fastchat/serve/inference.py#L91 - use
input_echo_lensliceoutput_idscausesoutputcontain part of the token in question, like***** ASSISTANT:。
https://github.com/lm-sys/FastChat/blob/34c585ced4c0b30d2264c519f626b4c5cb924216/fastchat/serve/inference.py#L178
Example：
max_new_tokens = 2048
Q：请介绍清华大学
truncate input_ids： an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: 请介绍清华大学 ASSISTANT:
output after slice： 华大学 ASSISTANT: 清华大学是中国著名的综合性研究型大学之一，位于北京市。该校成立于1911年，最初是清政府建立的留美预备学校，后来发展成为国立清华大学。清华大学是中国最早的高等学府之一，也是中国现代教育的重要奠基者之一。\n\n清华大学在多个学科领域都拥有很高的学术水平和知名度，其中工程、计算机、经济、管理等专业尤为著名。该校拥有一流的教学和研究设施，包括国家实验室、国家工程实验室、国家工程研究中心等。此外，清华大学还拥有多个国家级科研机构，如清华大学国家技术创新研究院、清华大学国家金融研究院等。\n\n清华大学的校园占地面积广阔，环境优美，是中国最美丽的大学校园之一。该校拥有多个校区，其中最著名的是清华大学本部，位于北京市海淀区清华路。清华大学的校园文化氛围浓厚，学生生活丰富多彩，学校也非常重视学生的全面发展和社会责任感的培养。\n\n清华大学是中国高等教育的重要标志之一，也是国际知名的高等学府之一。该校的毕业生在中国乃至世界各地都享有很高的声誉，许多毕业生成为了各个领域的领军人物。</s>
- Should consideration be given to moving
output_ids = list(input_ids)afterinput_ids = input_ids[-max_src_len:] - or drop
- 8, What does-8mean?
https://github.com/lm-sys/FastChat/blob/34c585ced4c0b30d2264c519f626b4c5cb924216/fastchat/serve/inference.py#L89
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lm-sys/FastChat
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
-
documentation good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100