Add `multipart_chunksize` / `multipart_threshold` to the `s3` remote config schema

Open Beginner friendly
#11,034 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python
Domain
cloud

Research direction

Start in dvc/config_schema.py, inspecting the s3 block and compare its accepted keys with the aliases in dvc_s3/init.py. Add schema coverage for the two multipart settings, then run the relevant configuration validation tests; done means values such as 64MiB load without an extra-key error and reach the plugin.

Written by the indexing model from the issue text.

Description

Problem

dvc-s3 already maps multipart_chunksize and multipart_threshold into boto3's TransferConfig via _TRANSFER_CONFIG_ALIASES (see dvc_s3/__init__.py lines 59–60, 117), but dvc/config_schema.py's s3 block (lines ~165–193) doesn't list either key. Result: any attempt to set them is rejected at config-load time, before the plugin ever sees them.

$ dvc remote modify --local codap-dvc multipart_chunksize 64MiB
ERROR: configuration error - config file error: extra keys not allowed @ data['remote']['PROFILE']['multipart_chunksize']

Reproduced on DVC 3.66.1 and 3.67.1.

Why this matters

Scaleway Object Storage caps multipart uploads at 1000 parts (vs AWS S3's 10,000). With boto3's default 8 MiB part size, the maximum object size is 8 MiB × 1000 = 8 GiB. Any file larger than that fails with:

ERROR: failed to transfer '<hash>' - [Errno 22] Part number must be an integer between 1 and 1000, inclusive: An error occurred (InvalidArgument) when calling the UploadPart operation

The intended fix — multipart_chunksize 64MiB, lifting the ceiling to 64 GiB — is correctly handled by dvc-s3 once the value reaches the plugin, but the config schema blocks it from ever getting there.

This is not Scaleway-specific in any architectural sense; the same gap blocks any S3-compatible backend with stricter multipart limits, and blocks legitimate tuning on AWS itself for very large objects.

Proposed fix

Add the two keys to the s3 block in dvc/config_schema.py:

"s3": {
    # ...existing keys...
    "read_timeout": Coerce(int),
    "connect_timeout": Coerce(int),
    "multipart_threshold": str,    # human-readable size, e.g. "64MiB"
    "multipart_chunksize": str,    # human-readable size, e.g. "64MiB"
    Optional("verify", default=False): Bool,
    **REMOTE_COMMON,
},

dvc-s3's _split_s3_config already calls human_readable_to_bytes on these two keys, so accepting str is correct. Optionally, max_queue_size and max_concurrent_requests (also in the alias map but not the schema) could be added in the same change — same gap, same rationale.

Workaround

For users blocked today, content-addressed objects can be uploaded directly via rclone (or any S3 client) with a sufficient chunk_size, written to the same <bucket>/files/md5/<2chars>/<rest> path DVC uses. dvc push becomes a HEAD-check no-op for those objects; dvc pull works transparently from any other machine since the path layout is the contract, not the upload mechanism. This isn't ideal — every operator with a >8 GiB object hits it — but it unblocks the immediate failure.

Environment

  • DVC: 3.66.1 (also reproduced on 3.67.1)
  • dvc-s3: 3.3.0
  • aiobotocore: 3.6.0, botocore: 1.43.0
  • Python 3.13.2, Linux
  • Remote: Scaleway Object Storage (S3-compatible, s3.fr-par.scw.cloud)
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from treeverse/dvc

All issues in treeverse/dvc

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.