Python: every dependency `.pyc` shipped to Lambda is stale on arrival (zip entries are dated `new Date(0)`), so each cold start recompiles all of them. One env var fixes it: −40% to −47% improvement to cold start

Đọc packages/serverless/lib/plugins/python/lib/pip.js và theo dõi cách dockerEnv được áp dụng cho quá trình cài đặt Docker cũng như cách tiến trình không dùng Docker được khởi chạy. Giữ nguyên SOURCE_DATE_EPOCH do người dùng cung cấp, đồng thời cung cấp giá trị mặc định cho cả hai đường đi, sau đó xác minh rằng bytecode của các dependency đã cài đặt sử dụng checked-hash invalidation và rằng hành vi hiện có của requirements cache đã được hiểu rõ.

Issue description

What happens

With the built-in Python requirements support (Framework 4.42.0, dockerizePip: non-linux, runtime python3.14), pip compiles each dependency to a .pyc at install time, and those files are packaged. They use CPython's default, timestamp-based invalidation: each .pyc records its source's mtime and size, and is only used while they still match.

The packager then writes every zip entry with a fixed date, so that the artifact hash is stable:

zip.append(file.data, { name, mode, date: new Date(0) })

1970 is before the zip format's epoch (1980), so the year underflows its 7-bit field and entries come out dated 2098-01-01 (1970 − 1980 = −10 → 118 → 2098). Whatever mtime Lambda gives the extracted sources, it is not the one recorded in the .pyc, so no dependency .pyc can ever validate. They are shipped, read, rejected and recompiled on every cold start, and because /var/task is read-only the result is never kept.

Nothing errors and nothing is logged. The function is just slow to start, for ever. Two sensible defaults (pip's and the zipper's) cancel each other out.

Hypothesis: why this has gone unnoticed

I don't think anyone made a mistake. Three sensible defaults meet badly, and the piece that fixes it arrived last:

• The Python plugin predates the fix. serverless-python-requirements dates from December 2016, its docker pip step from the same month. Hash-based .pyc files did not exist then: PEP 552 was written in September 2017 and shipped in Python 3.7 (June 2018). When the install step was written, there was no way to ship bytecode that survived a zip, and nothing has prompted anyone to revisit it since.
• PEP 552 deliberately kept timestamps as Python's default: a timestamp check is one stat, a hash check reads and hashes every source, and on an ordinary disk mtimes are honest. Hash-based files were aimed at reproducible builds, and pip simply calls compileall with the default.
• The zip dates are pinned on purpose, so that unchanged code is not redeployed (#13696 did the same for the esbuild path in July).
• It fails silently: no error, no log line, only a longer Init Duration, which gets blamed on Python or on heavy dependencies. The folklore runs the other way: slim strips .pyc files as dead weight, and in serverless-python-requirements#762 the cost of compiling was doubted rather than measured.

This is a hypothesis from the dates and the code, not something I have seen discussed; corrections welcome.

Evidence

Inspecting a real artifact from serverless package:

• 1,097 dependency .pyc files, all with flags 0 (timestamp-based), all recording the build time
• all 1,098 dependency .py entries dated 2098-01-01; 0 of 1,097 match

import zipfile, struct
z = zipfile.ZipFile(".serverless/<service>.zip")
for n in z.namelist():
    if n.endswith(".pyc"):
        flags, mtime, size = struct.unpack("<III", z.open(n).read(16)[4:16])
        print(n, "hash-based" if flags & 1 else f"timestamp {mtime}")  # compare with the .py entry's date_time
        break

Impact (measured)

Five production HTTP services (Flask or FastAPI, 825–1,731 dependency files, eu-west-2, python3.14). Cold starts forced with a burst of simultaneous requests and timed from outside. The only change between the columns is the workaround below; memory, code and region are the same.

Service	Memory	Dependency files	Cold start before	After	Change
A (FastAPI)	256 MB	1,686	4.47 s	2.39 s	−47%
B (Flask)	256 MB	1,105	3.70 s	1.96 s	−47%
C (FastAPI)	256 MB	825	3.08 s	1.86 s	−40%
D (Flask)	256 MB	1,731	2.34 s	1.27 s	−46%
E (FastAPI)	1024 MB	1,097	3.80 s (n=12)	2.10 s (n=6)	−45%

Medians; n is 2–8 per cell except where shown. The before and after ranges do not overlap for any service. Warm requests are unchanged (~50 ms). Locally, importing service E with the standard library's own bytecode intact: 928 ms with no usable dependency bytecode, 370 ms with checked-hash bytecode.

For comparison, quadrupling E's memory from 256 MB to 1024 MB (at 4x the price per ms) bought 13%.

Workaround (no hook, no plugin)

py_compile switches its default to hash-based invalidation (PEP 552, CHECKED_HASH) whenever SOURCE_DATE_EPOCH is set, and dockerEnv reaches the pip container:

custom:
  pythonRequirements:
    dockerizePip: non-linux
    dockerEnv:
      - SOURCE_DATE_EPOCH=315532800   # 1 Jan 1980. The value is irrelevant; only its presence matters

Two traps. dockerEnv only reaches pip when pip runs in Docker: on a Linux host with dockerizePip: non-linux, pip is spawned directly and inherits the shell, so export the variable there instead. And the static requirements cache is keyed on the sha256 of requirements.txt plus the architecture (getRequirementsWorkingPath), not on dockerEnv, so after adding the setting the old bytecode keeps being served from cache with no sign that anything is wrong: clear serverless-python-requirements in the user cache directory once. After this, 98–100% of dependency .pyc files in each of our six artifacts are hash-based and match their sources (importlib.util.source_hash); the few exceptions are files pip does not compile.

Suggested fix

In packages/serverless/lib/plugins/python/lib/pip.js, set SOURCE_DATE_EPOCH for the install step by default, both in the docker run environment (next to where dockerEnv is applied) and in the environment of the non-Docker spawn, unless the user has already set it. Or, equivalently, run python -m compileall --invalidation-mode checked-hash over the installed requirements as a post-install step.

• checked-hash costs one read and one hash per module at import and can never run stale code; a mismatched source is simply recompiled, exactly as today. The saving of unchecked-hash over it was within noise here, so there is no reason to take that risk.
• The fixed zip dates are load-bearing for deploy change-detection (#13696 pinned them on the esbuild path for exactly that reason), so the fix belongs at compile time, not in the zipper. Hash-based .pyc files help that goal too: a timestamp-based .pyc embeds the build time, so two builds of identical requirements produce different artifact bytes; hash-based ones are byte-identical.
• This needs Python 3.7+ (PEP 552), which is every runtime Lambda still supports. Older Pythons ignore the variable and compile as before, so a default is harmless there: checked on 3.6.15 (timestamp .pyc, unchanged) and 3.7.17 (checked-hash).
• The uv path deserves a look too: uv pip install does not compile bytecode unless given --compile-bytecode, so those artifacts presumably ship none at all and pay the same cost. I have not measured that path.

At the very least, a note in the Python packaging docs would help. People reach for more memory or provisioned concurrency to fix a cost that is one environment variable.

Prior discussion

I searched issues, pull requests and Discussions here, and the archived plugin's issues, for pyc, bytecode, SOURCE_DATE_EPOCH, compileall, invalidation-mode, PYTHONDONTWRITEBYTECODE, zip timestamps and Python cold starts, and found no report of this.

Related

• aws/aws-lambda-builders#924: the same fault in SAM (its zips carry 1980 dates), open
• serverless/serverless-python-requirements#762 (repository now archived): asks to "make use of compiled .pyc files"; does not identify that the shipped ones are already stale, and proposes a userland hook, which the setting above replaces
• https://blog.frank-mich.com/recipe-reducing-cold-start-with-precompiled-python-bytecode-in-serverless/

Context

No response