Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

PTE and PTD serialization drop stricter alignment requests for deduplicated named data

Chiusa
  • Qualcosa non va?
#23,302 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@JacobSzwejbka ci sta già lavorando.

Dal 1/10/2026.

  • #23517 di @JacobSzwejbka — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug module: exir triaged
🐛 Describe the bug

When two named-data entries reference identical bytes but request different alignments, both PTE and PTD serialization retain only the first entry's alignment. The final location can violate the second entry's requirement. Reversing registration order changes the result.

The NamedDataStore contract says duplicate blobs combine alignment requirements using LCM. The merged serialization design, #8835, also explicitly describes combining requests from two backends before combining with segment_alignment.

Minimal PTD reproduction

Use the normal Python export environment with flatc available. If the wheel's bundled executable is not on PATH, set FLATC_EXECUTABLE to its executorch/data/bin/flatc.

from executorch.exir._serialize._named_data_store import NamedDataStore
from executorch.exir._serialize.data_serializer import DataPayload
from executorch.extension.flat_tensor.serialize.serialize import (
    FlatTensorSerializer, FlatTensorHeader, _deserialize_to_flat_tensor,
)

store = NamedDataStore()
# Keeps segment_base_offset at 512, isolating alias handling from base alignment.
store.add_named_data("prefix" + "p" * 160, b"x", alignment=128)
store.add_named_data("weak", b"abcd", alignment=16)
store.add_named_data("strong", b"abcd", alignment=256)
out = store.get_named_data_store_output()
raw = bytes(FlatTensorSerializer().serialize(DataPayload(out.buffers, out.pte_data)))
header = FlatTensorHeader.from_bytes(raw[8:])
metadata = _deserialize_to_flat_tensor(raw[:header.flatbuffer_offset + header.flatbuffer_size])
entry = next(entry for entry in metadata.named_data if entry.key == "strong")
segment = metadata.segments[entry.segment_index]
absolute = header.segment_base_offset + segment.offset
print(header.segment_base_offset, segment.offset, absolute, absolute % 256)
assert absolute % 256 == 0

Expected: the shared segment satisfies both alignment requests.

Actual stdout and full exception traceback (directory prefix shortened):

512 128 640 128
Traceback (most recent call last):
  File "issue_repro.py", line 20, in <module>
    assert absolute % 256 == 0
           ^^^^^^^^^^^^^^^^^^^
AssertionError
PTE confirmation and controls

The equivalent PTE path through serialize_pte_binary produces the same offsets. Both _extract_named_data implementations create AlignedData using the first alias's alignment and do not combine later aliases' requirements:

Controls were tested separately for both formats, all with segment_base_offset == 512:

  • Same bytes, alignment 16 first then 256: one shared segment at absolute offset 640; the 256-byte request fails
  • Same bytes, alignment 256 first then 16: one shared segment at absolute offset 768; both requests pass
  • Different bytes: separate segments satisfy both requests in either order
  • Serialized payload bytes remain correct, with the expected deduplicated segment count

The padded prefix name makes the segment base itself satisfy the largest request, isolating alias handling. Without this control, base-offset alignment can independently affect the result.

Twelve focused checks produced 4 failures and 8 passes. The failures are the expected-LCM assertion and final-offset assertion for weak-first aliases, in each format. The unchanged NamedDataStore and FlatTensor suites passed (28 tests and 5 subtests), as did the two existing PTE named-data tests. The full repository suite was not run.

Existing coverage tests different alignment requests at the store level, or serialized aliases with the same alignment, but does not test this combination:

This demonstrates a serialized alignment-contract violation. No model, device, backend execution or inference failure was tested.

Is the documented LCM contract still intended? Would a focused fix that combines alignment requirements per deduplicated buffer in both serializers, with these regression tests, be appropriate?

Investigation and reproducer prepared with OpenAI Codex.

Versions

Tested using the official ExecuTorch 1.5.1 wheel on Linux x86_64. The installed exir/_serialize/_program.py, _named_data_store.py, data_serializer.py and extension/flat_tensor/serialize/serialize.py were verified byte-for-byte against main commit a0f09ad35b91141940995ce60561412d2b9a6756.

Additional versions: flatbuffers 25.12.19, flatc 24.3.25, pytest 9.1.1.

Output from python -m torch.utils.collect_env:

<frozen runpy>:128: RuntimeWarning: 'torch.utils.collect_env' found in sys.modules after import of package 'torch.utils', but prior to execution of 'torch.utils.collect_env'; this may result in unpredictable behaviour
Collecting environment information...
PyTorch version: 2.14.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 13 (trixie) (x86_64)
GCC version: (Debian 14.2.0-19) 14.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.41

Python version: 3.12.14 (main, Aug 25 2026, 14:00:49) [Clang 22.1.3 ] (64-bit runtime)
Python platform: Linux-6.18.44-x86_64-with-glibc2.41
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A

CPU:
lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or directory

Versions of relevant libraries:
[pip3] executorch==1.5.1
[pip3] numpy==2.5.3
[pip3] torch==2.14.0+cpu
[pip3] torchao==0.18.0
[conda] Could not collect

cc @JacobSzwejbka @angelayi @nil-is-all

Lingua principale
Python
Stelle
5.1k
Fork
1.2k
Merge medio
2g 10h
PR unite (30g)
632

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pytorch/executorch

Tutte le issue di pytorch/executorch

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.