[Proposal] MiniCPM adapter (MiniCPMForCausalLM) — I'm working on it

Abierto
  • ¿Algo no está bien?
#1,992 1 comentario 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@pablocs116 ya está trabajando en esto.

Desde el 6/11/2026.

  • #1994 de @pablocs116 — abierto

Evaluación

Este issue todavía no se ha evaluado.

Descripción

Proposal

Add a TransformerBridge architecture adapter for MiniCPM (MiniCPMForCausalLM, model_type: minicpm, loaded via trust_remote_code). I'm working on this and will open a PR against dev.

Motivation

MiniCPMForCausalLM is listed in the architecture gap registry (71 models, ~928k downloads, relevancy 69.7) and isn't in the adapter factory yet. No existing issue or PR covers it.

Pitch

Scope: dense MiniCPM checkpoints only.

Validation checkpoints:

  • openbmb/MiniCPM4-0.5B (primary)
  • openbmb/MiniCPM-1B-sft-bf16 (if it fits my hardware; CPU only)

The architecture is Llama-shaped (RMSNorm, RoPE/LongRoPE, GQA, gated SiLU MLP, tied embeddings) with three µP-style scalings, which map onto existing Granite-adapter machinery:

  • scale_emb: embedding output multiplier
  • scale_depth / sqrt(num_hidden_layers): residual-branch multiplier
  • hidden_size / dim_model_base: hidden state divided before lm_head (logit scaling)

Under transformers v5 the remote code needs a few loading-time patches, handled in the adapter's prepare_loading / prepare_model the same way the Baichuan and InternLM2 adapters do:

  1. is_torch_fx_available was removed from transformers.utils.import_utils and the remote code still imports it.
  2. _tied_weights_keys is the v4 list form; v5 expects a dict.
  3. The rotary buffers (inv_freq, cos_cached, sin_cached) are non-persistent and come out of v5 loading uninitialized. The model still runs but produces garbage, so they're recomputed after load.
  4. forward raises if use_cache=True without a Cache object.

With those fixes, MiniCPM4-0.5B loads and generates correctly on transformers 5.8.1.

Out of scope:

  • MiniCPM4-8B / MiniCPM4.1-8B: sparse (InfLLM-v2) attention
  • MiniCPM3 (MiniCPM3ForCausalLM): a different class with MLA attention
  • MiniCPM-S and BitCPM variants: not validated
Alternatives

Waiting for upstream to update the remote code for v5. The patches are small and isolated, so I'd rather not block on that.

Checklist
  • I have checked that there is no similar issue in the repo (required)
Lenguaje dominante
Python
Estrellas
3.9k
Forks
708
Merge medio
1 d 4 h
PR fusionados (30 d)
87

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de TransformerLensOrg/TransformerLens

Todos los issues de TransformerLensOrg/TransformerLens

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.