Multi-vector textual inversion uses only the first vector when the token has uppercase letters (transformers v5)
维护者通常 1 天内回复
评估
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 75/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
调研方向
先阅读 _maybe_convert_prompt 和相关的 maybe_convert_prompt 测试。使用包含大写字母的多向量 textual inversion token 重现问题,然后检查已注册的 token 是如何匹配和替换的。完成标准是:使用含大写字母的 token 的 prompt 会扩展为基础 token 和所有编号向量;通过回归测试验证混合大小写的注册和 prompt 文本两种情况。
由索引模型根据 Issue 内容生成。
描述
Describe the bug
With transformers v5, a multi-vector textual inversion embedding only contributes its first vector when its token contains uppercase letters. This happens both when the token is loaded that way (token="EasyNegative") and when it is written that way in the prompt (Neg for a token loaded as neg). No error or warning is raised.
maybe_convert_prompt should expand the token to <token> <token>_1 ... <token>_n before tokenizing. In v5, CLIPTokenizer matches added tokens case-insensitively and tokenize() returns them lowercased. But _maybe_convert_prompt looks the result up in tokenizer.added_tokens_encoder and replaces it in the prompt case-sensitively, so nothing gets expanded. With transformers 4.57.1, tokenize() returned the token as registered and the expansion worked.
Proposed fix: in _maybe_convert_prompt, find the registered token through its lowercased form and replace it case-insensitively. Happy to open a PR with a test if this approach works for you.
Reproduction
import torch
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained("hf-internal-testing/tiny-sd-pipe", safety_checker=None)
dim = pipe.text_encoder.config.hidden_size
pipe.load_textual_inversion(torch.randn(3, dim), token="MyToken") # adds MyToken, MyToken_1, MyToken_2
pipe.load_textual_inversion(torch.randn(3, dim), token="neg") # adds neg, neg_1, neg_2
for text in ["MyToken", "neg", "Neg"]:
prompt = pipe.maybe_convert_prompt(text, pipe.tokenizer) # what the pipeline passes to the tokenizer
print(f"{text!r:10} -> {pipe.tokenizer.tokenize(prompt)}")
Output on main:
'MyToken' -> ['mytoken']
'neg' -> ['neg', 'neg_1', 'neg_2']
'Neg' -> ['neg']
Expected: all three vectors in every case, e.g. 'MyToken' -> ['mytoken', 'mytoken_1', 'mytoken_2'].
Context
This affects embeddings made for A1111, which are referred to by their capitalized file name in prompts. For example, EasyNegative has 8 vectors. With the SD 1.5 tokenizer and the token loaded as EasyNegative, the prompt EasyNegative, blurry is expanded to 8 tokens with transformers 4.57.1 and stays a single token with 5.22.2.
Logs
No error or warning is raised.
System Info
- 🤗 Diffusers version: 0.44.0.dev0 (main, 1dabc292d)
- Platform: macOS-26.6.2-arm64-arm-64bit
- Running on Google Colab?: No
- Python version: 3.12.11
- PyTorch version (GPU?): 2.15.0 (False)
- Huggingface_hub version: 2.4.0
- Transformers version: 5.22.2
- Accelerate version: 1.15.0
- PEFT version: 0.21.2
- Safetensors version: 0.8.0
- xFormers version: not installed
- Accelerator: Apple M5 Pro
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
Who can help?
@sayakpaul
- 主要语言
- Python
- 星标
- 34.9k
- 派生
- 7.5k
- 平均合并
- 3 天 16 小时
- 30 天内合并 PR
- 65
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
huggingface/diffusers 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 84/100
huggingface/diffusers#15363 ·
维护者通常 1 天内回复
-
`ZImageImg2ImgPipeline` / `ZImageInpaintPipeline`: explicit height/width different from the input image raises RuntimeError可能已有人在做 @chessing234 今天认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 70/100
huggingface/diffusers#15337 · 2 条评论 ·
维护者通常 1 天内回复
-
TCDScheduler advances the generator on the final step of a strength-truncated custom schedule可能已有人在做 @SIDDARTHAREDDY8 于 5 天前认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 74/100
huggingface/diffusers#15288 · 2 条评论 ·
维护者通常 1 天内回复
-
unload_textual_inversion(token) leaves the extra tokens of a multi-vector embedding behind, so it can't be loaded again可能已有人在做 @sohailshk 于 11 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
huggingface/diffusers#15211 · 3 条评论 ·
维护者通常 1 天内回复
-
set_ip_adapter_scale(float) raises "Cannot assign 1 scale_configs to 2 IP-Adapter" when several IP-Adapters are loaded可能已有人在做 @sohailshk 于 11 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
huggingface/diffusers#15210 · 1 条评论 ·
维护者通常 1 天内回复
查看 huggingface/diffusers 的全部 Issue
相似的 Issue
-
[dbt] exported dataset keys reference physical columns可能已有人在做 @WilliamKelley 今天认领。 未关闭needs-triage
难度 2/5 1-3 小时 新手友好度 68/100
apache/ossie-converters#251 ·
维护者通常 2 天内回复
-
cli difficulty: easy enhancement
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
desec-io/desec-stack#1312 ·
维护者通常 14 天内回复
-
needs-triage
难度 2/5 1-3 小时 新手友好度 82/100
aws/aws-durable-execution-sdk-python#977 ·
维护者通常 1 天内回复
-
needs-triage
难度 1/5 1 小时以内 新手友好度 88/100
flashinfer-ai/flashinfer#9803 ·
维护者通常 1 天内回复