[Bug] ChatGLMTokenizer special token property getters and setters are broken
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 84/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python, pytorch
- Domain
- machine-learning
Research direction
Open src/diffusers/pipelines/kolors/tokenizer.py and inspect ChatGLMTokenizer's pad_token, unk_token, and eos_token properties. Reproduce the reported default values and assignments such as tokenizer.pad_token = "<custom_pad>". Done means the getters return the correct defaults and reflect values assigned through their setters.
Written by the indexing model from the issue text.
Description
Describe the bug
In ChatGLMTokenizer (src/diffusers/pipelines/kolors/tokenizer.py), the property getters for pad_token, unk_token, and eos_token hardcode return values (e.g. pad_token returns "<unk>" instead of "<pad>"). Furthermore, assigning new special token values (e.g. tokenizer.pad_token = "<custom_pad>") has no effect on the property getter because the custom setters update private attributes (_pad_token) while the getters ignore them or use hardcoded strings.
Reproduction
from diffusers.pipelines.kolors import ChatGLMTokenizer
# Instantiating ChatGLMTokenizer or viewing properties
# tokenizer.pad_token returns '<unk>' instead of '<pad>'
# Setting tokenizer.pad_token = '<custom_pad>' does not update tokenizer.pad_token
System Info
- Diffusers version: 0.36.0.dev0
- Python version: 3.11
- PyTorch version: 2.x
Who can help?
@yiyixuxu @asomoza @DN6
- Dominant language
- Python
- Stars
- 34.6k
- Forks
- 7.4k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 76
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/diffusers
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
huggingface/diffusers#14837 ·
-
bug needs-env-info pipelines
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
huggingface/diffusers#14794 ·
-
bug needs-code-example needs-env-info pipelines
Difficulty 1/5 Under an hour Newbie friendliness 86/100
huggingface/diffusers#14780 · 1 comment ·
-
bug pipelines
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/diffusers#14769 · 1 comment ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
huggingface/diffusers#14639 ·
All issues in huggingface/diffusers
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·