Generator for chunks of content stream

Aperta
#96 2 commenti 0 reazioni 1 assegnatario Vedi su GitHub

@jmoldow ci sta già lavorando.

Dal 14/12/2015.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

enhancement

Today, there are two options for downloading a file:

  • File.content(), which requests the entire file at once, and loads it all into memory.
  • File.download_to(), which requests the file in chunks, and dumps it into a stream (file handle or another instance of io.IOBase).

The first option always loads the full contents into memory. The second option always loads the full contents into memory except in the case where the stream is a file handle. Additionally, both options always download the full file before handing control back to the caller.

There should be a third option for download, which only downloads and loads individual chunks at a time, and is a generator and therefore hands control back to the caller for each chunk.

I've implemented this both locally and in a comment on #93, and it looks something like this:

from contextlib import contextmanager, closing

class File(Item):
    ...

    # A contextmanager so that we can force the HTTP stream to be closed when we're done with it.
    @contextmanager
    def content_chunks(self):
        url = self.get_url('content')
        with closing(self._session.get(url, expect_json_response=False, stream=True)) as box_response:
            yield box_response.network_response.response_as_stream().stream(decode_content=True)

    # We can redefine download_to() in terms of content_chunks().
    def download_to(self, writeable_stream):
        with self.content_chunks() as chunks:
            for chunk in chunks:
                writeable_stream.write(chunk)

# Usage example 1
with my_file.content_chunks() as chunks:
    for chunk in chunks:
        # Handle chunk

# Usage example 2
with my_file.content_chunks() as chunks:
    # Handle chunks

This could be added as-is, but I haven't created a PR yet because I wanted to think more about #87. I was wondering if there might be a better way to expose streams in the abstract Network interface. Right now we have a generic response_as_stream() method on NetworkResponse, but it isn't really generic because:

  • It requires stream=True to be passed to request() on Network, which isn't documented because it is specific to the requests library.
  • After calling response_as_stream(), you must call its stream() method, which isn't documented because it is specific to the requests library.

We could just add this as-is, because it doesn't technically add any additional dependencies on requests that weren't already added via download_to().

Lingua principale
Python
Stelle
460
Fork
223
Merge medio
14h 18m
PR unite (30g)
21

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di box/box-python-sdk

Tutte le issue di box/box-python-sdk

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.