[python] Generated output differs between identical runs without --embed-auditing-info
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- build-system
Research direction
Start by reading the three causes in the issue and inspect src/nunavut/lang/py/templates/Namespace.j2, src/nunavut/lang/py/__init__.py around filter_pickle, and src/nunavut/_namespace.py around get_nested_types(). Check the Python generation tests for an existing output comparison pattern. Done means repeated runs without --embed-auditing-info produce identical output, with regression coverage for the timestamp, gzip metadata, and namespace ordering.
Written by the indexing model from the issue text.
Description
Summary
Running the same nnvg --target-language=py command twice produces different files, even without --embed-auditing-info. Generated output is supposed to be identical when that flag isn't given (#324 reversed the flag so that reproducible output is the default). C output is identical between runs; Python output is not.
Reproduce
On main at ed853f9, in ghcr.io/opencyphal/toxic:tx26.4.1:
PRDT=submodules/public_regulated_data_types/uavcan
nnvg $PRDT --target-language=py --outdir a
nnvg $PRDT --target-language=py --outdir b
diff -rq a b | wc -l # 244 of 245 files differ (only nunavut_support.py matches)
The same two runs with --target-language=c produce 0 differing files out of 176. C++ was not checked.
Causes
There are three independent causes. All three must be fixed for the output to be reproducible.
- The namespace file's timestamp ignores
--embed-auditing-info(affects all 69__init__.py).src/nunavut/lang/py/templates/Namespace.j2:8writes# Generated at: {{ now_utc }} UTCunconditionally. Python's type modules (py/templates/base.j2) and the C and C++base.j2templates write it only inside{%- if nunavut.embed_auditing_info %}. - The embedded model blob records the time (affects all 175 type modules).
filter_pickle(src/nunavut/lang/py/__init__.py:355) callsgzip.compress(pickle.dumps(x, protocol=4)). Without anmtimeargument, gzip writes the current time into its header, so each module's_MODEL_constant changes on every run. Decoding the blobs from both runs gives identical pickled payloads. Only the 4-byte gzipMTIMEheader field differs (for example1792505892against1792505915). Nothing else in the type modules differs. - Namespace file import order is nondeterministic (affects 30 of 68
uavcan__init__.py). Ignoring the timestamp line, 30 namespace files still differ, always in the order of theirfrom … import …lines and never in which types they list.Namespace.get_nested_types()(src/nunavut/_namespace.py:904) returns types in insertion order, and insertion order follows the order the DSDL files were read (set iteration plus parallel parse-job completion).
Expected
Without --embed-auditing-info, running the same command twice produces byte-for-byte identical Python output, as it does for C. With the flag, volatile metadata such as the timestamp may appear, as it does in the other languages.
Possible fixes
- Wrap the
Generated atline inpy/templates/Namespace.j2in{%- if nunavut.embed_auditing_info %}, matchingpy/templates/base.j2. - Pass
mtime=0togzip.compressinfilter_pickle. Consumers (_restore_constant_) only callgzip.decompress, which ignores the field. - Emit namespace members in a stable order, e.g. sorted by full name and version, in
get_nested_types()or in the template.
A regression test could generate the same input twice into two directories, without --embed-auditing-info, and assert that the trees are identical, for every target language.
Found while working on #403. It doesn't block #403, which needs semantic idempotence, not byte-for-byte reproducibility.
- Dominant language
- Python
- Stars
- 48
- Forks
- 26
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 9
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from OpenCyphal/nunavut
-
build and test dependencies
Difficulty 4/5 3-5 days Newbie friendliness 55/100
OpenCyphal/nunavut#419 ·
Maintainers usually reply within 1 day
-
build and test C++ dependencies
Difficulty 4/5 3-5 days Newbie friendliness 50/100
OpenCyphal/nunavut#418 ·
Maintainers usually reply within 1 day
-
build and test Python
Difficulty 4/5 3-5 days Newbie friendliness 45/100
OpenCyphal/nunavut#416 ·
Maintainers usually reply within 1 day
-
C C++ Python
Difficulty 4/5 3-5 days Newbie friendliness 52/100
OpenCyphal/nunavut#414 · 1 comment ·
Maintainers usually reply within 1 day
-
feature under consideration
Difficulty 5/5 Over a week Newbie friendliness 32/100
OpenCyphal/nunavut#413 ·
Maintainers usually reply within 1 day
All issues in OpenCyphal/nunavut
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
py-econometrics/pyfixest#2072 ·
Maintainers usually reply within 1 day
-
~planning docs
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
Kromatic-Innovation/athenaeum#3084 ·
Maintainers usually reply within 1 day
-
Bosch Room Thermostat II number entities offer -500 to 3000 °C setpoint limits and a -32768 to 32767 °C outdoor temperaturePossibly taken @zigpy-review-bot claimed this today. Openbot: code audit
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
zigpy/zha-device-handlers#5769 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 Half a day Newbie friendliness 80/100
R0Y00/Xbox2Windows#254 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Kludex/python-multipart#375 ·
Maintainers usually reply within 1 day