Class creation with many annotated classes in one module is superlinear on 3.14 (annotationlib FORWARDREF path, suspected: module globals copied per call)

オープン
  • 何か問題がありますか？
#161,912 コメント 3 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
backend

調査の方向性

Start with Lib/annotationlib.py and inspect call_annotate_function and _StringifierDict.__init__, the suspected path identified in the issue. Reproduce the import timings with the supplied gen.py script on Python 3.14 and compare them with 3.13. The goal is to find and fix the cause of superlinear TypedDict class creation so import time scales roughly linearly with class count.

索引モデルが issue の本文から書いたものです。

説明

performance stdlib

Bug description

Importing a module that defines tens of thousands of TypedDict classes takes 25-31 s on Python 3.14.7, against 1.6-2.0 s on 3.13.16, for the same source (reproduction script below). Time grows faster than linearly with the number of classes in the module: 3.14.7 needs 0.45 s for 3k classes, 2.8 s for 10k and 25.6 s for 30k; 3.13.16 needs 0.15 s, 0.45 s and 1.6 s (variant without forward references).

It is the same with typing.TypedDict and typing_extensions.TypedDict, so it is not specific to typing_extensions. It does not need unresolved forward references: a variant whose annotations are all resolvable (List[int]) took 34 s, as long as the variant with forward references.

Real-world case: the client generated by Prisma Client Python (prisma/types.py, 10.7 MB, 30,546 classes, 30,344 of them TypedDict) takes about 41 s to import on 3.14. cProfile attributes 38 of 39 s to annotationlib.call_annotate_function. The generated module is a legitimate, if large, use of annotations; on 3.13 the same import is a fraction of a second.

Suspected cause

In Lib/annotationlib.py (3.14 branch), call_annotate_function with Format.FORWARDREF builds

namespace = {**annotate.__builtins__, **annotate.__globals__}
...
globals = _StringifierDict(namespace, globals=annotate.__globals__, ...)

and _StringifierDict.__init__ calls super().__init__(namespace), which copies the dict again. Each call therefore copies the module globals twice. For a module that defines N classes, the globals hold O(N) entries by the time later classes are built, so building N classes costs O(N^2). typing_extensions.TypedDict.__new__ (and typing.TypedDict on 3.14) calls this for every class.

cProfile of the 30,000-class forward-reference module on 3.14.7 (32.7 s in total): 30,000 calls of call_annotate_function take 17.0 s of own time, _StringifierDict.__init__ 4.7 s, and typing.TypedDict.__new__ 7.2 s of own time, which I cannot explain from the source. A micro-benchmark of copying a 30,000-entry dict gives 0.17 ms, so two copies per call explain roughly 8 s of the total at most; I have not confirmed that they are the dominant cost.

Reproduction

# gen.py - writes mod_N.py with N TypedDict classes, then times its import
import sys, time, importlib, pathlib

n = int(sys.argv[1])
fwd = len(sys.argv) > 2 and sys.argv[2] == "fwd"   # fwd: each class refers to the next class
lines = ["from typing import TypedDict, List"]
for i in range(n):
    ref = f"'C{i+1}'" if fwd and i + 1 < n else "int"
    lines += [f"class C{i}(TypedDict, total=False):", f"    a: List[int]", f"    b: {ref}", ""]
path = pathlib.Path(f"mod_{n}_{'fwd' if fwd else 'plain'}.py")
path.write_text("\n".join(lines))
sys.path.insert(0, ".")
t = time.perf_counter()
importlib.import_module(path.stem)
print(f"{sys.version.split()[0]} n={n} {'fwd' if fwd else 'plain'}: {time.perf_counter() - t:.2f}s")

Run for n in 3000, 10000, 30000, with and without fwd, on 3.13 and 3.14 (run twice, the first run includes .pyc compilation). Measured with this script (30,000 classes, the second run uses the .pyc): 3.13.16 1.6 s plain / 2.0 s fwd, 3.14.7 25.6 s plain / 31.4 s fwd; the .pyc run is 0.7-1.1 s on 3.13 and 25.6-30.7 s on 3.14, so compilation is not the cost. In an earlier synthetic test 3.12.14 took about 1.5-1.8 s and 3.15.0rc3 about 23 s (a release candidate; faster, but does not remove the problem).

Expected behaviour

Class creation time roughly linear in the number of classes, as on 3.13.

Your environment

  • CPython 3.14.7 on Linux x86_64 (also measured: 3.12.14, 3.13.16, 3.15.0rc3)
  • typing.TypedDict and typing_extensions.TypedDict give the same result

Questions for maintainers

  • Is the per-call copy of module globals in the FORWARDREF path intended, or could the namespace be a ChainMap / a view instead of a copy?
  • Does the same cost appear for ordinary (non-TypedDict) classes with many annotated attributes in a large module? Not tested.
主要言語
Python
スター
78.2k
フォーク
36.6k
平均マージ
19時間 57分
マージ済み PR（30日）
630

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

python/cpython のほかの issue

python/cpython の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。