Cloud Storage SDK: Attempting to upload files larger than Java heap space results in OOM

Issue Details

Required: Provide a detailed description of the issue. Any additional information helps with figuring out the issue.

Example

I am seeing {ISSUE} when running X, Y... after doing Z. This behavior started after the X library upgrade a few weeks ago ...

Environment

Required: Provide details about your environment. If relevant, please also provide details about your GCP environment. The environment details MUST be filled out.
Provide the relevant details about your environment:

• OS: Fedora 44
• Java Version and JDK Vendor: Open JDK 17.0.17
• (If using GraalVM) GraalVM Version: N/A

(Optional) If deployed on GCP, provide the relevant details about your GCP deployment environment:

Storage test bench 0.54.0 running in podman

Dependencies

Required: Provide a list of the dependencies that are being used. If unable to provide the list, please list the relevant libraries shown below. One of the two sections below MUST be filled out.

Provide the versions of the relevant libraries:
- Libraries-Bom (com.google.cloud:libraries-bom): N/A
- (If not using Libraries-Bom) Client Libraries:
- Gax (gax / gax-grpc / gax-httpjson): 2.83.0
- Auth (google-auth-library-oauth2-http / google-auth-library-credentials): 1.50.0
- Any other relevant Java SDK dependencies:
- Google-Http-Java-Client (com.google.http-client:google-http-client): 2.2.0

Reproducer

Required: Provide a reproducer and the steps needed to reproduce this issue locally. If unable to provide a reproducer, please provide code snippets to help reproduce the issue. One of the two sections below MUST be filled out.

Provide as many code snippets as possible:

Example

Create transfer manager:

public GoogleStorageClient(StorageClientConfig clientConfig) {
        config = new GoogleStorageClientConfig(clientConfig);
        GoogleCredentials credentials = config.googleCredentials;
        if(credentials instanceof AnonymousGoogleCredentials) {
            storage = StorageOptions.newBuilder().setCredentials(credentials).setHost(clientConfig.getEndpointUrl())
            .setTransportOptions(HttpTransportOptions.newBuilder().setConnectTimeout(GoogleCloudConfig.GOOGLE_REQUEST_TIMEOUT).build())
            .setProjectId(config.projectId).build().getService();
        } else {
            storage = StorageOptions.newBuilder().setCredentials(credentials).setProjectId(config.projectId).setTransportOptions(HttpTransportOptions.newBuilder().setConnectTimeout(GoogleCloudConfig.GOOGLE_REQUEST_TIMEOUT).build()).build().getService();
        }
        transferManager = TransferManagerConfig.newBuilder().setAllowDivideAndConquerDownload(true).setAllowParallelCompositeUpload(true).setStorageOptions(storage.getOptions()).build().getService();
    }

Same class

public uploadFiles(File[] sourceFiles, ) {
   ParallelUploadConfig parallelUploadConfig = ParallelUploadConfig.newBuilder().setBucketName(bucketName).build();
   List<UploadResult> results =
           transferManager.uploadFiles(files, parallelUploadConfig).getUploadResults();
  
}

Create a client and attempt to upload a large file, larger than the JVM internal memory. I tested with a 3.2 GB file with 512 MB heap. The upload fails at the uploadFiles call with

Java heap spacecom.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
	at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1383)
	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1376)
	at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:53)
	at com.google.cloud.storage.transfermanager.UploadJob.getUploadResults(UploadJob.java:57)
	at com.ibm.cmus.google_cloud_storage.utils.client.GoogleStorageClient.uploadFiles(GoogleStorageClient.java:436)

I develop a tool which provides methods for interacting with various storage vendors (i.e. s3). I don't see other transfer managers struggling with memory and I am able to upload the same file without issues. While the simple fix is to increase heap space, as that size is obviously quite small, it is enough for other vendors to upload the same file.

Logs and Stack Trace

Required: Provide logs that showcase the error. This will help show the flow of the application and help narrow down the cause. Additionally, provide a stack trace of the error if possible.

FYI, the below link is broken, the troubleshooting guide appears to have been removed sometime in the last 2 years.
The Java SDK has a troubleshooting guide for enabling logs. This contains information regarding client-server communication, request and response details, and logging in dependency libraries. If using this guide, please obfuscate any private information (bearer tokens, request and response params, etc).

Additionally, please provide a stack trace of the error seen:

Exception

Java heap spacecom.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
	at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1383)
	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1376)
	at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:53)
	at com.google.cloud.storage.transfermanager.UploadJob.getUploadResults(UploadJob.java:57)
	at com.ibm.cmus.google_cloud_storage.utils.client.GoogleStorageClient.uploadFiles(GoogleStorageClient.java:436)

Behavior

Optional: Any additional information about the behavior of the error is helpful to debug.

Behavioral Questions:

• When did the issue begin? Is this behavior related to any dependency version upgrade? - I have not tested files of this size on older versions
• Is this behavior flaky? Or is this consistently seen in production? - This is consistently seen. The memory requirements for uploading files grow with file size.
• Is this behavior related to the volume of data? - No, but it is related to the size of the file.