Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

PTE and PTD serialization drop stricter alignment requests for deduplicated named data

Đã đóng
  • Có gì không ổn?
#23,302 1 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@JacobSzwejbka đang làm issue này rồi.

Từ ngày 1/10/2026.

  • #23517 của @JacobSzwejbka — đang mở

Đánh giá

Issue này chưa được đánh giá.

Mô tả

bug module: exir triaged
🐛 Describe the bug

When two named-data entries reference identical bytes but request different alignments, both PTE and PTD serialization retain only the first entry's alignment. The final location can violate the second entry's requirement. Reversing registration order changes the result.

The NamedDataStore contract says duplicate blobs combine alignment requirements using LCM. The merged serialization design, #8835, also explicitly describes combining requests from two backends before combining with segment_alignment.

Minimal PTD reproduction

Use the normal Python export environment with flatc available. If the wheel's bundled executable is not on PATH, set FLATC_EXECUTABLE to its executorch/data/bin/flatc.

from executorch.exir._serialize._named_data_store import NamedDataStore
from executorch.exir._serialize.data_serializer import DataPayload
from executorch.extension.flat_tensor.serialize.serialize import (
    FlatTensorSerializer, FlatTensorHeader, _deserialize_to_flat_tensor,
)

store = NamedDataStore()
# Keeps segment_base_offset at 512, isolating alias handling from base alignment.
store.add_named_data("prefix" + "p" * 160, b"x", alignment=128)
store.add_named_data("weak", b"abcd", alignment=16)
store.add_named_data("strong", b"abcd", alignment=256)
out = store.get_named_data_store_output()
raw = bytes(FlatTensorSerializer().serialize(DataPayload(out.buffers, out.pte_data)))
header = FlatTensorHeader.from_bytes(raw[8:])
metadata = _deserialize_to_flat_tensor(raw[:header.flatbuffer_offset + header.flatbuffer_size])
entry = next(entry for entry in metadata.named_data if entry.key == "strong")
segment = metadata.segments[entry.segment_index]
absolute = header.segment_base_offset + segment.offset
print(header.segment_base_offset, segment.offset, absolute, absolute % 256)
assert absolute % 256 == 0

Expected: the shared segment satisfies both alignment requests.

Actual stdout and full exception traceback (directory prefix shortened):

512 128 640 128
Traceback (most recent call last):
  File "issue_repro.py", line 20, in <module>
    assert absolute % 256 == 0
           ^^^^^^^^^^^^^^^^^^^
AssertionError
PTE confirmation and controls

The equivalent PTE path through serialize_pte_binary produces the same offsets. Both _extract_named_data implementations create AlignedData using the first alias's alignment and do not combine later aliases' requirements:

Controls were tested separately for both formats, all with segment_base_offset == 512:

  • Same bytes, alignment 16 first then 256: one shared segment at absolute offset 640; the 256-byte request fails
  • Same bytes, alignment 256 first then 16: one shared segment at absolute offset 768; both requests pass
  • Different bytes: separate segments satisfy both requests in either order
  • Serialized payload bytes remain correct, with the expected deduplicated segment count

The padded prefix name makes the segment base itself satisfy the largest request, isolating alias handling. Without this control, base-offset alignment can independently affect the result.

Twelve focused checks produced 4 failures and 8 passes. The failures are the expected-LCM assertion and final-offset assertion for weak-first aliases, in each format. The unchanged NamedDataStore and FlatTensor suites passed (28 tests and 5 subtests), as did the two existing PTE named-data tests. The full repository suite was not run.

Existing coverage tests different alignment requests at the store level, or serialized aliases with the same alignment, but does not test this combination:

This demonstrates a serialized alignment-contract violation. No model, device, backend execution or inference failure was tested.

Is the documented LCM contract still intended? Would a focused fix that combines alignment requirements per deduplicated buffer in both serializers, with these regression tests, be appropriate?

Investigation and reproducer prepared with OpenAI Codex.

Versions

Tested using the official ExecuTorch 1.5.1 wheel on Linux x86_64. The installed exir/_serialize/_program.py, _named_data_store.py, data_serializer.py and extension/flat_tensor/serialize/serialize.py were verified byte-for-byte against main commit a0f09ad35b91141940995ce60561412d2b9a6756.

Additional versions: flatbuffers 25.12.19, flatc 24.3.25, pytest 9.1.1.

Output from python -m torch.utils.collect_env:

<frozen runpy>:128: RuntimeWarning: 'torch.utils.collect_env' found in sys.modules after import of package 'torch.utils', but prior to execution of 'torch.utils.collect_env'; this may result in unpredictable behaviour
Collecting environment information...
PyTorch version: 2.14.0+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 13 (trixie) (x86_64)
GCC version: (Debian 14.2.0-19) 14.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.41

Python version: 3.12.14 (main, Aug 25 2026, 14:00:49) [Clang 22.1.3 ] (64-bit runtime)
Python platform: Linux-6.18.44-x86_64-with-glibc2.41
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A

CPU:
lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or directory

Versions of relevant libraries:
[pip3] executorch==1.5.1
[pip3] numpy==2.5.3
[pip3] torch==2.14.0+cpu
[pip3] torchao==0.18.0
[conda] Could not collect

cc @JacobSzwejbka @angelayi @nil-is-all

Ngôn ngữ chính
Python
Star
5.1k
Fork
1.2k
Merge trung bình
2 ngày 9 giờ
Pull request đã merge (30 ngày)
632

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của pytorch/executorch

Tất cả issue của pytorch/executorch

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.