Multi-vector textual inversion uses only the first vector when the token has uppercase letters (transformers v5)

未关闭 适合新手
#15,313 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@sunhouy 已经在做这个了。

开始于 2026年11月18日。

  • #15335 来自 @sunhouy —— 未关闭

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
75/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python, pytorch

调研方向

先阅读 _maybe_convert_prompt 和相关的 maybe_convert_prompt 测试。使用包含大写字母的多向量 textual inversion token 重现问题,然后检查已注册的 token 是如何匹配和替换的。完成标准是:使用含大写字母的 token 的 prompt 会扩展为基础 token 和所有编号向量;通过回归测试验证混合大小写的注册和 prompt 文本两种情况。

由索引模型根据 Issue 内容生成。

描述

Describe the bug

With transformers v5, a multi-vector textual inversion embedding only contributes its first vector when its token contains uppercase letters. This happens both when the token is loaded that way (token="EasyNegative") and when it is written that way in the prompt (Neg for a token loaded as neg). No error or warning is raised.

maybe_convert_prompt should expand the token to <token> <token>_1 ... <token>_n before tokenizing. In v5, CLIPTokenizer matches added tokens case-insensitively and tokenize() returns them lowercased. But _maybe_convert_prompt looks the result up in tokenizer.added_tokens_encoder and replaces it in the prompt case-sensitively, so nothing gets expanded. With transformers 4.57.1, tokenize() returned the token as registered and the expansion worked.

Proposed fix: in _maybe_convert_prompt, find the registered token through its lowercased form and replace it case-insensitively. Happy to open a PR with a test if this approach works for you.

Reproduction
import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained("hf-internal-testing/tiny-sd-pipe", safety_checker=None)
dim = pipe.text_encoder.config.hidden_size
pipe.load_textual_inversion(torch.randn(3, dim), token="MyToken")  # adds MyToken, MyToken_1, MyToken_2
pipe.load_textual_inversion(torch.randn(3, dim), token="neg")  # adds neg, neg_1, neg_2

for text in ["MyToken", "neg", "Neg"]:
    prompt = pipe.maybe_convert_prompt(text, pipe.tokenizer)  # what the pipeline passes to the tokenizer
    print(f"{text!r:10} -> {pipe.tokenizer.tokenize(prompt)}")

Output on main:

'MyToken'  -> ['mytoken']
'neg'      -> ['neg', 'neg_1', 'neg_2']
'Neg'      -> ['neg']

Expected: all three vectors in every case, e.g. 'MyToken' -> ['mytoken', 'mytoken_1', 'mytoken_2'].

Context

This affects embeddings made for A1111, which are referred to by their capitalized file name in prompts. For example, EasyNegative has 8 vectors. With the SD 1.5 tokenizer and the token loaded as EasyNegative, the prompt EasyNegative, blurry is expanded to 8 tokens with transformers 4.57.1 and stays a single token with 5.22.2.

Logs
No error or warning is raised.
System Info
  • 🤗 Diffusers version: 0.44.0.dev0 (main, 1dabc292d)
  • Platform: macOS-26.6.2-arm64-arm-64bit
  • Running on Google Colab?: No
  • Python version: 3.12.11
  • PyTorch version (GPU?): 2.15.0 (False)
  • Huggingface_hub version: 2.4.0
  • Transformers version: 5.22.2
  • Accelerate version: 1.15.0
  • PEFT version: 0.21.2
  • Safetensors version: 0.8.0
  • xFormers version: not installed
  • Accelerator: Apple M5 Pro
  • Using GPU in script?: No
  • Using distributed or parallel set-up in script?: No
Who can help?

@sayakpaul

主要语言
Python
星标
34.9k
派生
7.5k
平均合并
3 天 16 小时
30 天内合并 PR
65

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

huggingface/diffusers 的其他 Issue

查看 huggingface/diffusers 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。