Allow chunked uploads (and maybe even streaming uploads)
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 50/100
Research direction
Start in dropbox_client.py around lines 533-539, where request bodies are restricted to bytes, and compare that behavior with the linked Requests streaming and chunked-upload documentation. Determine how the SDK should accept file-like objects or generators while documenting retry and rewind behavior; done means the chosen upload forms are supported without the current bytes-only restriction.
Written by the indexing model from the issue text.
Description
Why is this feature valuable to you? Does it solve a problem you're having?
The requests library allows both streaming and chunked uploads (see https://requests.readthedocs.io/en/latest/user/advanced/#streaming-uploads and https://requests.readthedocs.io/en/latest/user/advanced/#chunk-encoded-requests). This has two benefits:
- It is sufficient to only load small parts of a file into memory before upload.
- It is possible to limit bandwidth usage by using a generator that provides chunks at a limited rate.
The Dropbox API of course already requires upload sessions (files/upload_session_start, files/upload_session_append and files/upload_session_finish) to upload files > 150 MB. However, this approach by itself does not replace chunked or streaming uploads because:
- The request body should be ideally >= 4 MB to reduce the total number of API calls (both for efficiency and to not exhaust data transport API call limits).
- Bandwidth control will be very coarse when performed on chunks of 4 MB compared for example 2 kB.
- Memory usage will still be larger compared to 1 kB or 2 kB chunks, especially for parallel uploads.
Describe the solution you'd like
Requests supports streaming uploads by passing a file-like object as the request body and chunked uploads by passing a generator as the request body. However, the Python SDK explicitly prevents both by requiring the request body to be of type bytes:
It would be good to either completely drop this limitation, with appropriate warnings in the doc string, or at least allow chunked uploads (where requests handles retry / rewind logic) even when disallowing streaming uploads.
Describe alternatives you've considered
Not at present.
- Dominant language
- Python
- Stars
- 983
- Forks
- 331
- Avg merge
- 3m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dropbox/dropbox-sdk-python
-
question
Difficulty 4/5 3-5 days Newbie friendliness 35/100
dropbox/dropbox-sdk-python#585 · 4 comments ·
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 35/100
dropbox/dropbox-sdk-python#491 · 1 comment ·
-
enhancement
Difficulty 3/5 1-2 days Newbie friendliness 35/100
dropbox/dropbox-sdk-python#482 · 1 comment · 2 reactions ·
-
question
Difficulty 5/5 Over a week Newbie friendliness 25/100
dropbox/dropbox-sdk-python#476 · 3 comments ·
-
bug
Difficulty 5/5 Over a week Newbie friendliness 20/100
dropbox/dropbox-sdk-python#475 · 1 comment ·
All issues in dropbox/dropbox-sdk-python
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·