Class creation with many annotated classes in one module is superlinear on 3.14 (annotationlib FORWARDREF path, suspected: module globals copied per call)

Aperta
  • Qualcosa non va?
#161,912 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend

Direzione di ricerca

Start with Lib/annotationlib.py and inspect call_annotate_function and _StringifierDict.__init__, the suspected path identified in the issue. Reproduce the import timings with the supplied gen.py script on Python 3.14 and compare them with 3.13. The goal is to find and fix the cause of superlinear TypedDict class creation so import time scales roughly linearly with class count.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

performance stdlib

Bug description

Importing a module that defines tens of thousands of TypedDict classes takes 25-31 s on Python 3.14.7, against 1.6-2.0 s on 3.13.16, for the same source (reproduction script below). Time grows faster than linearly with the number of classes in the module: 3.14.7 needs 0.45 s for 3k classes, 2.8 s for 10k and 25.6 s for 30k; 3.13.16 needs 0.15 s, 0.45 s and 1.6 s (variant without forward references).

It is the same with typing.TypedDict and typing_extensions.TypedDict, so it is not specific to typing_extensions. It does not need unresolved forward references: a variant whose annotations are all resolvable (List[int]) took 34 s, as long as the variant with forward references.

Real-world case: the client generated by Prisma Client Python (prisma/types.py, 10.7 MB, 30,546 classes, 30,344 of them TypedDict) takes about 41 s to import on 3.14. cProfile attributes 38 of 39 s to annotationlib.call_annotate_function. The generated module is a legitimate, if large, use of annotations; on 3.13 the same import is a fraction of a second.

Suspected cause

In Lib/annotationlib.py (3.14 branch), call_annotate_function with Format.FORWARDREF builds

namespace = {**annotate.__builtins__, **annotate.__globals__}
...
globals = _StringifierDict(namespace, globals=annotate.__globals__, ...)

and _StringifierDict.__init__ calls super().__init__(namespace), which copies the dict again. Each call therefore copies the module globals twice. For a module that defines N classes, the globals hold O(N) entries by the time later classes are built, so building N classes costs O(N^2). typing_extensions.TypedDict.__new__ (and typing.TypedDict on 3.14) calls this for every class.

cProfile of the 30,000-class forward-reference module on 3.14.7 (32.7 s in total): 30,000 calls of call_annotate_function take 17.0 s of own time, _StringifierDict.__init__ 4.7 s, and typing.TypedDict.__new__ 7.2 s of own time, which I cannot explain from the source. A micro-benchmark of copying a 30,000-entry dict gives 0.17 ms, so two copies per call explain roughly 8 s of the total at most; I have not confirmed that they are the dominant cost.

Reproduction

# gen.py - writes mod_N.py with N TypedDict classes, then times its import
import sys, time, importlib, pathlib

n = int(sys.argv[1])
fwd = len(sys.argv) > 2 and sys.argv[2] == "fwd"   # fwd: each class refers to the next class
lines = ["from typing import TypedDict, List"]
for i in range(n):
    ref = f"'C{i+1}'" if fwd and i + 1 < n else "int"
    lines += [f"class C{i}(TypedDict, total=False):", f"    a: List[int]", f"    b: {ref}", ""]
path = pathlib.Path(f"mod_{n}_{'fwd' if fwd else 'plain'}.py")
path.write_text("\n".join(lines))
sys.path.insert(0, ".")
t = time.perf_counter()
importlib.import_module(path.stem)
print(f"{sys.version.split()[0]} n={n} {'fwd' if fwd else 'plain'}: {time.perf_counter() - t:.2f}s")

Run for n in 3000, 10000, 30000, with and without fwd, on 3.13 and 3.14 (run twice, the first run includes .pyc compilation). Measured with this script (30,000 classes, the second run uses the .pyc): 3.13.16 1.6 s plain / 2.0 s fwd, 3.14.7 25.6 s plain / 31.4 s fwd; the .pyc run is 0.7-1.1 s on 3.13 and 25.6-30.7 s on 3.14, so compilation is not the cost. In an earlier synthetic test 3.12.14 took about 1.5-1.8 s and 3.15.0rc3 about 23 s (a release candidate; faster, but does not remove the problem).

Expected behaviour

Class creation time roughly linear in the number of classes, as on 3.13.

Your environment

  • CPython 3.14.7 on Linux x86_64 (also measured: 3.12.14, 3.13.16, 3.15.0rc3)
  • typing.TypedDict and typing_extensions.TypedDict give the same result

Questions for maintainers

  • Is the per-call copy of module globals in the FORWARDREF path intended, or could the namespace be a ChainMap / a view instead of a copy?
  • Does the same cost appear for ordinary (non-TypedDict) classes with many annotated attributes in a large module? Not tested.
Lingua principale
Python
Stelle
78.2k
Fork
36.6k
Merge medio
19h 57m
PR unite (30g)
630

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di python/cpython

Tutte le issue di python/cpython

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.