Generator for chunks of content stream
@jmoldow ci sta già lavorando.
Dal 14/12/2015.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
Today, there are two options for downloading a file:
File.content(), which requests the entire file at once, and loads it all into memory.File.download_to(), which requests the file in chunks, and dumps it into a stream (filehandle or another instance ofio.IOBase).
The first option always loads the full contents into memory. The second option always loads the full contents into memory except in the case where the stream is a file handle. Additionally, both options always download the full file before handing control back to the caller.
There should be a third option for download, which only downloads and loads individual chunks at a time, and is a generator and therefore hands control back to the caller for each chunk.
I've implemented this both locally and in a comment on #93, and it looks something like this:
from contextlib import contextmanager, closing
class File(Item):
...
# A contextmanager so that we can force the HTTP stream to be closed when we're done with it.
@contextmanager
def content_chunks(self):
url = self.get_url('content')
with closing(self._session.get(url, expect_json_response=False, stream=True)) as box_response:
yield box_response.network_response.response_as_stream().stream(decode_content=True)
# We can redefine download_to() in terms of content_chunks().
def download_to(self, writeable_stream):
with self.content_chunks() as chunks:
for chunk in chunks:
writeable_stream.write(chunk)
# Usage example 1
with my_file.content_chunks() as chunks:
for chunk in chunks:
# Handle chunk
# Usage example 2
with my_file.content_chunks() as chunks:
# Handle chunks
This could be added as-is, but I haven't created a PR yet because I wanted to think more about #87. I was wondering if there might be a better way to expose streams in the abstract Network interface. Right now we have a generic response_as_stream() method on NetworkResponse, but it isn't really generic because:
- It requires
stream=Trueto be passed torequest()onNetwork, which isn't documented because it is specific to the requests library. - After calling
response_as_stream(), you must call itsstream()method, which isn't documented because it is specific to the requests library.
We could just add this as-is, because it doesn't technically add any additional dependencies on requests that weren't already added via download_to().
- Lingua principale
- Python
- Stelle
- 460
- Fork
- 223
- Merge medio
- 14h 18m
- PR unite (30g)
- 21
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di box/box-python-sdk
-
enhancement
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
box/box-python-sdk#196 ·
-
bug enhancement
box/box-python-sdk#1314 · 6 commenti · 5 assegnatari ·
-
enhancement
box/box-python-sdk#1154 · 3 commenti · 5 assegnatari ·
-
Asynchronous IO Support Apertaenhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 25/100
box/box-python-sdk#1155 · 1 commento · 4 reazioni ·
-
enhancement
box/box-python-sdk#1157 · 1 commento · 6 assegnatari ·
Tutte le issue di box/box-python-sdk
Issue simili
-
triage/confirmed
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
agentscope-ai/agentscope#2775 ·
-
comp/desktop P3 type/bug
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 92/100
NousResearch/hermes-agent#118866 ·
-
bug
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 90/100
apache/cloudstack#14222 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100