`compile_and_save` needs twice the engine size on disk while it saves to ExecuTorch

Đang mở
  • Có gì không ổn?
#5,041 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 3 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
18/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python, pytorch
Lĩnh vực
data, devtools

Hướng nghiên cứu

The save path is in compile_and_save in the Torch-TensorRT Python package, which writes each built engine to a temporary file before the ExecuTorch copy; the repro script in the issue measures peak disk use and is the check for done. The fix spans two projects: the ExecuTorch file-writing code that would need to punch holes in owned file-backed data after each chunk is copied, and the Torch-TensorRT hand-off of offloaded engine files. Start by reading how engine files are written and passed to ExecuTorch, then confirm the disk peak drops to about one copy.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

What is wrong

compile_and_save writes each engine to a temporary file as soon as it is built. With output_format="executorch", the save then copies each file into the .pte (or .ptd). The temporary files are deleted only when the call returns. So at the end of the save, every engine is on disk twice.

For a large single-engine program this decides whether the export fits. #5026 lets the caller move the files to another disk, which helps, but the space is still needed.

What this blocks

No policy in pytorch/executorch#24353 is listed as failing because of this. It costs disk.
Any export with output_format="executorch" needs free space for twice its engines, so a large single-engine program needs twice its size free. That can decide whether it fits on a Jetson or a DGX Spark.
A fix lets an export fit when the free disk is only a little larger than the program.

Repro
import os, shutil, tempfile, threading, time
import torch, torch_tensorrt

work = tempfile.mkdtemp()
os.environ["TMPDIR"] = tempfile.tempdir = os.path.join(work, "tmp")
os.makedirs(tempfile.tempdir)
out = os.path.join(work, "model.pte")

def used(path):
    return sum(os.stat(os.path.join(r, f)).st_blocks * 512 for r, _, fs in os.walk(path) for f in fs)

peak, stop = [0], threading.Event()
def watch():
    while not stop.is_set():
        peak[0] = max(peak[0], used(work))
        time.sleep(0.2)

model = torch.nn.Sequential(*[torch.nn.Linear(8192, 8192, bias=False) for _ in range(32)]).cuda().bfloat16().eval()
x = torch.randn(4, 8192, device="cuda", dtype=torch.bfloat16)
ep = torch.export.export(model, (x,))
threading.Thread(target=watch, daemon=True).start()
torch_tensorrt.compile_and_save(ep, out, output_format="executorch", arg_inputs=[x], min_block_size=1)
stop.set()
print(f"program {os.path.getsize(out) / 2**30:.2f} GiB, disk peak {peak[0] / 2**30:.2f} GiB")
shutil.rmtree(work)
Actual
program 4.00 GiB, disk peak 8.00 GiB
Expected

The peak is about one copy. A proposal in 2 small parts:

  1. ExecuTorch: file-backed data that the caller marks as owned is consumed by the write. After each chunk is copied, that range of the source is released (fallocate with FALLOC_FL_PUNCH_HOLE, supported on ext4, XFS and btrfs), and the file is removed at the end. Elsewhere it copies as today.
  2. Torch-TensorRT: hand each offloaded engine file to ExecuTorch as owned data, since nothing reads it after the save.
Versions

torch 2.15.0 (CUDA 13.2), torch_tensorrt 2.15.0, executorch 1.6.0, TensorRT 11.4.0.106. Script run on H100, driver 580.126.09 (btrfs).

Related: #4863 (keeping built engines on disk until the save, which is where the temporary files come from), #5026.

Ngôn ngữ chính
Python
Star
3k
Fork
421
Merge trung bình
4 ngày 7 giờ
Pull request đã merge (30 ngày)
62

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của pytorch/TensorRT

Tất cả issue của pytorch/TensorRT

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.