Class creation with many annotated classes in one module is superlinear on 3.14 (annotationlib FORWARDREF path, suspected: module globals copied per call)

未关闭
  • 哪里不对？
#161,912 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
48/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python
领域
backend

调研方向

Start with Lib/annotationlib.py and inspect call_annotate_function and _StringifierDict.__init__, the suspected path identified in the issue. Reproduce the import timings with the supplied gen.py script on Python 3.14 and compare them with 3.13. The goal is to find and fix the cause of superlinear TypedDict class creation so import time scales roughly linearly with class count.

由索引模型根据 Issue 内容生成。

描述

performance stdlib

Bug description

Importing a module that defines tens of thousands of TypedDict classes takes 25-31 s on Python 3.14.7, against 1.6-2.0 s on 3.13.16, for the same source (reproduction script below). Time grows faster than linearly with the number of classes in the module: 3.14.7 needs 0.45 s for 3k classes, 2.8 s for 10k and 25.6 s for 30k; 3.13.16 needs 0.15 s, 0.45 s and 1.6 s (variant without forward references).

It is the same with typing.TypedDict and typing_extensions.TypedDict, so it is not specific to typing_extensions. It does not need unresolved forward references: a variant whose annotations are all resolvable (List[int]) took 34 s, as long as the variant with forward references.

Real-world case: the client generated by Prisma Client Python (prisma/types.py, 10.7 MB, 30,546 classes, 30,344 of them TypedDict) takes about 41 s to import on 3.14. cProfile attributes 38 of 39 s to annotationlib.call_annotate_function. The generated module is a legitimate, if large, use of annotations; on 3.13 the same import is a fraction of a second.

Suspected cause

In Lib/annotationlib.py (3.14 branch), call_annotate_function with Format.FORWARDREF builds

namespace = {**annotate.__builtins__, **annotate.__globals__}
...
globals = _StringifierDict(namespace, globals=annotate.__globals__, ...)

and _StringifierDict.__init__ calls super().__init__(namespace), which copies the dict again. Each call therefore copies the module globals twice. For a module that defines N classes, the globals hold O(N) entries by the time later classes are built, so building N classes costs O(N^2). typing_extensions.TypedDict.__new__ (and typing.TypedDict on 3.14) calls this for every class.

cProfile of the 30,000-class forward-reference module on 3.14.7 (32.7 s in total): 30,000 calls of call_annotate_function take 17.0 s of own time, _StringifierDict.__init__ 4.7 s, and typing.TypedDict.__new__ 7.2 s of own time, which I cannot explain from the source. A micro-benchmark of copying a 30,000-entry dict gives 0.17 ms, so two copies per call explain roughly 8 s of the total at most; I have not confirmed that they are the dominant cost.

Reproduction

# gen.py - writes mod_N.py with N TypedDict classes, then times its import
import sys, time, importlib, pathlib

n = int(sys.argv[1])
fwd = len(sys.argv) > 2 and sys.argv[2] == "fwd"   # fwd: each class refers to the next class
lines = ["from typing import TypedDict, List"]
for i in range(n):
    ref = f"'C{i+1}'" if fwd and i + 1 < n else "int"
    lines += [f"class C{i}(TypedDict, total=False):", f"    a: List[int]", f"    b: {ref}", ""]
path = pathlib.Path(f"mod_{n}_{'fwd' if fwd else 'plain'}.py")
path.write_text("\n".join(lines))
sys.path.insert(0, ".")
t = time.perf_counter()
importlib.import_module(path.stem)
print(f"{sys.version.split()[0]} n={n} {'fwd' if fwd else 'plain'}: {time.perf_counter() - t:.2f}s")

Run for n in 3000, 10000, 30000, with and without fwd, on 3.13 and 3.14 (run twice, the first run includes .pyc compilation). Measured with this script (30,000 classes, the second run uses the .pyc): 3.13.16 1.6 s plain / 2.0 s fwd, 3.14.7 25.6 s plain / 31.4 s fwd; the .pyc run is 0.7-1.1 s on 3.13 and 25.6-30.7 s on 3.14, so compilation is not the cost. In an earlier synthetic test 3.12.14 took about 1.5-1.8 s and 3.15.0rc3 about 23 s (a release candidate; faster, but does not remove the problem).

Expected behaviour

Class creation time roughly linear in the number of classes, as on 3.13.

Your environment

  • CPython 3.14.7 on Linux x86_64 (also measured: 3.12.14, 3.13.16, 3.15.0rc3)
  • typing.TypedDict and typing_extensions.TypedDict give the same result

Questions for maintainers

  • Is the per-call copy of module globals in the FORWARDREF path intended, or could the namespace be a ChainMap / a view instead of a copy?
  • Does the same cost appear for ordinary (non-TypedDict) classes with many annotated attributes in a large module? Not tested.
主要语言
Python
星标
78.2k
派生
36.6k
平均合并
19 小时 57 分钟
30 天内合并 PR
630

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

python/cpython 的其他 Issue

查看 python/cpython 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。