Improving representative benchmarks for typing ecosystem

Open
#105 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
performance

Research direction

Start by reviewing the three benchmark areas described in the issue and the existing pydantic benchmarks linked there. Compare those options with the current pyperformance benchmark suite, then clarify which cases and benchmark programs should be included. Done means a representative benchmark scope is agreed and the selected benchmarks are upstreamed.

Written by the indexing model from the issue text.

Description

Due to a current lack of representative macrobenchmarks, it is very difficult to decide on whether complex accelerators for some parts of typing are worth implementing in the future. Hence, I'm trying to upstream some benchmarks into pyperformance.

IMO, there are three main areas:

  1. Performance of static type checkers implemented in Python (e.g. mypy). (Fixed by #102)
  2. Performance of programs using types at runtime (e.g. pydantic, attrs, etc.).
  3. Runtime overhead of typed code vs fully untyped code.

For case 2, I plan to use one of pydantic's benchmarks here https://github.com/samuelcolvin/pydantic/tree/master/benchmarks, installed without compiled binaries.

Case 3 is very tricky because there are so many ways to use typing. I don't know how often people use certain features, whether they type-hint inside tight loops, etc. So I'm struggling to find a good benchmark. An idea: grabbing one of the existing pyperformance benchmarks, fully type-hinting it, then comparing the performance delta may work.

CC @JelleZijlstra, I would greatly appreciate hearing your opinion on this (especially for case 3). Maybe I can post this on typing-sig too if I need more help.

Afterword:
All 3 cases benefit from general CPython optimizations. But usually only 3. benefits greatly from typing module-only optimizations (with 1. maybe not improving much if at all, depending on implementation).

Dominant language
Python
Stars
1k
Forks
203
Avg merge
1h 20m
Merged PRs (30d)
2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from python/pyperformance

All issues in python/pyperformance

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.