unload_textual_inversion(token) leaves the extra tokens of a multi-vector embedding behind, so it can't be loaded again

未关闭 适合新手
  • 哪里不对？
#15,211 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
72/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python, pytorch
领域
machine-learning

调研方向

首先阅读 textual inversion 的加载和卸载方法，以及它们使用的 maybe_convert_prompt 查找逻辑；issue 没有指定文件路径或测试。使用小型 Stable Diffusion pipeline 和双向量 embedding 复现。完成标准是：卸载基础 token 时，也会删除其带后缀的 token 和 embedding 行，从而可以再次加载同一个 embedding。

由索引模型根据 Issue 内容生成。

描述

Describe the bug

load_textual_inversion registers a multi-vector embedding as <token>, <token>_1, ..., <token>_{n-1}, and maybe_convert_prompt expands <token> into all of them. unload_textual_inversion("<token>") only removes <token> itself. The _1 ... _{n-1} tokens and their embedding rows stay in the tokenizer and text encoder, and loading the same embedding again fails with Multi-vector Token [...] already in tokenizer vocabulary.

Users only pass <token> to load_textual_inversion, so they have no reason to know about the suffixed tokens. #7856 hit this with an SDXL negative embedding and was closed with a workaround (list every suffixed token), but the behaviour is still there on main.

I'd like to send a PR for this: when tokens is given, also remove its <token>_1, <token>_2, ... tokens, using the same lookup maybe_convert_prompt uses. Happy to adjust if you prefer a different approach.

Reproduction

Shown with the tiny test pipeline so it runs in seconds; the loader code is the same for SD1.5 and SDXL embeddings.

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained("hf-internal-testing/tiny-stable-diffusion-torch", safety_checker=None)

embedding = torch.randn(2, 32)  # a 2-vector embedding (this tiny text encoder has hidden size 32)
pipe.load_textual_inversion(embedding, token="<neg>")
pipe.unload_textual_inversion("<neg>")
print([t for t in pipe.tokenizer.get_added_vocab() if t.startswith("<neg>")])  # ['<neg>_1']

pipe.load_textual_inversion(embedding, token="<neg>")  # ValueError
Context

I ran into this while testing loading and unloading of multi-vector textual inversion embeddings.

Logs
ValueError: Multi-vector Token ['<neg>', '<neg>_1'] already in tokenizer vocabulary. Please choose a different token name or remove the ['<neg>', '<neg>_1'] and embedding from the tokenizer and text encoder.
System Info
- 🤗 Diffusers version: 0.42.0.dev0 (main @ 84532a058)
- Platform: macOS-26.6.2-arm64-arm-64bit
- Running on Google Colab?: No
- Python version: 3.12.11
- PyTorch version (GPU?): 2.15.0 (False)
- Huggingface_hub version: 2.4.0
- Transformers version: 5.22.2
- Accelerate version: 1.15.0
- PEFT version: 0.21.2
- Safetensors version: 0.8.0
- xFormers version: not installed
- Accelerator: Apple M5 Pro
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
Who can help?

@sayakpaul

主要语言
Python
星标
34.9k
派生
7.5k
平均合并
2 天 16 小时
30 天内合并 PR
57

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

huggingface/diffusers 的其他 Issue

查看 huggingface/diffusers 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。