google-cloud-storage: Failed to handle doubly-gzipped files

Aperta
#18,423 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

  • #18430 di @ohmayr — chiusa senza merge

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
68/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
google-cloud, python
Ambito
cloud

Direzione di ricerca

Start by running the main.py reproduction and tracing the BlobReader read path, especially how it handles content and transfer encoding for the doubly-gzipped object. Add regression coverage for both raw_download values and verify that the second read returns 0, matching the expected results.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

api: storage priority: p3 type: bug
Determine this is the right repository
  • I determined this is the correct repository in which to report this bug.
Summary of the issue

When handling a doubly gzipped file (the first test file in the code below), the reader fails to reach the end of file.

Image
API client name and version

google-cloud-storage 3.14.1

Reproduction steps: code

file: main.py

from google.cloud.storage import Blob
from google.cloud.storage import Bucket
from google.cloud.storage import Client

from google.cloud.storage.fileio import BlobReader

client = Client("apache-beam-testing")
bucket = Bucket(client=client, name="apache-beam-samples")

test_files = [
    "textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.default.1k.txt",
    "textio/textio-test-data.default.1k.txt.gz",
    "textio/textio-test-data.gzip-local.1k.txt.gz",
]

for raw_download in [False, True]:
  print(f"-- raw_download={raw_download} --")
  for test_file in test_files:
    blob = Blob(test_file, bucket=bucket)
    reader = BlobReader(blob, raw_download=raw_download)
    a = reader.read(16777216)
    b = reader.read(16777216)
    print(f"[{test_file}] first_read:{len(a)}, second_read:{len(b)}")
Reproduction steps: supporting files

Run the code directly.

Reproduction steps: actual results
-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:101000   <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:76478   <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0
Reproduction steps: expected results
-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0
OS & version + platform

MacOS, Linux

Python environment

Python 3.12

Python dependencies

google-cloud-storage 3.14.1

Additional context

No response

Lingua principale
Python
Stelle
5.4k
Fork
1.8k
Merge medio
2g 20h
PR unite (30g)
140

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di googleapis/google-cloud-python

Tutte le issue di googleapis/google-cloud-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.