Streaming Google storage upload
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Read the limitations described in google-cloud-ruby issue #1997 and inspect the linked gcs_stream_upload wrapper, using its streaming examples as the starting point. Done means Google storage uploads can write streamed data without retaining the full contents in memory, including data produced through IO.copy_stream.
Written by the indexing model from the issue text.
Description
I am trying to solve the case of uploading files in a streaming fashion without keeping the full contents in memory
I have previously implemented the same feature in https://github.com/aws/aws-sdk-ruby/pull/1711
As identified in https://github.com/googleapis/google-cloud-ruby/issues/1997
There are some limitations in google-api-client which hard codes the use of the content length header
I have written a minimal wrapper library that monkey patches it here which should serve as a good source of inspiration: https://github.com/Fonsan/gcs_stream_upload
require 'gcs_stream_upload'
storage = Google::Cloud::Storage.new(timeout: 5 * 60)
bucket = storage.bucket('bucket')
gcs_stream_upload = GCSStreamUpload.new(bucket)
gcs_stream_upload.upload('object') do |io|
IO.copy_stream(IO.popen('yes | head -n 10'), io)
end
# => Written "y\n" * 10
gcs_stream_upload.upload('object') do |io|
io << 'data'
io << 'dat2'
end
# => Written "datadat2"
- Dominant language
- Ruby
- Stars
- 2.9k
- Forks
- 888
- Avg merge
- 8m
- Merged PRs (30d)
- 241
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from googleapis/google-api-ruby-client
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
googleapis/google-api-ruby-client#28001 · 5 reactions ·
-
googleapis/google-api-ruby-client#27757 · 1 assignee ·
-
googleapis/google-api-ruby-client#27758 · 1 assignee ·
-
googleapis/google-api-ruby-client#27759 · 1 assignee ·
-
api: storage type: feature request
Difficulty 3/5 1-2 days Newbie friendliness 55/100
googleapis/google-api-ruby-client#27339 ·
All issues in googleapis/google-api-ruby-client
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
hanami/hanami-cli#449 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
resque/resque-scheduler#826 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100