Humongous allocation with CRT client when downloading chunks of files

Open
#6,566 7 comments 0 reactions 1 assignee View on GitHub

@debora-ito is already working on this.

Since Nov 14, 2025.

Assessment

This issue has not been assessed yet.

Description

bug crt-client
Describe the bug

When downloading a 4mb chunk from a large file, we were seeing humongous allocation from native code. CRT client was passing heap bytes byte[] bodyBytesIn to the caller.

Can we make it offheap?

Version: aws-crt:0.33.9

Lowering the part size helps to allocate smaller bytes, but still heap pressure.

We are using 8mb as part size. Will lowering it to 512kb affecting partial download or full file downloading performance?

class S3MetaRequestResponseHandlerNativeAdapter {

    int onResponseBody(byte[] bodyBytesIn, long objectRangeStart, long objectRangeEnd) {
        return this.responseHandler.onResponseBody(ByteBuffer.wrap(bodyBytesIn), objectRangeStart, objectRangeEnd);
    }
Regression Issue
  • Select this option if this issue appears to be a regression.
Expected Behavior

Offheap buffer should be used

Current Behavior

Heap buffer is used

Reproduction Steps

Download 4mb chunk from a large file.

        GetObjectRequest request = GetObjectRequest.builder()
                .bucket(bucket)
                .key(rkey)
                .range(String.format("bytes=%d-%d", pos, pos+size-1))
                .build();

        return s3Async.getObject(request, asyncResponseTransformerImp);
Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

aws-crt:0.33.9

JDK version used

JDK22

Operating System and version

Ubuntu 22.04 LTS. Also reproduced on Mac M1.

Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 17h
Merged PRs (30d)
39

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws/aws-sdk-java-v2

All issues in aws/aws-sdk-java-v2

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.