google-cloud-storage: Failed to handle doubly-gzipped files
I maintainer di solito rispondono entro 2 giorni
Nessuno ha ancora preso questa issue.
- #18430 di @ohmayr — chiusa senza merge
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Idoneità per principianti
- 68/100
Direzione di ricerca
Start by running the main.py reproduction and tracing the BlobReader read path, especially how it handles content and transfer encoding for the doubly-gzipped object. Add regression coverage for both raw_download values and verify that the second read returns 0, matching the expected results.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Determine this is the right repository
- I determined this is the correct repository in which to report this bug.
Summary of the issue
When handling a doubly gzipped file (the first test file in the code below), the reader fails to reach the end of file.
API client name and version
google-cloud-storage 3.14.1
Reproduction steps: code
file: main.py
from google.cloud.storage import Blob
from google.cloud.storage import Bucket
from google.cloud.storage import Client
from google.cloud.storage.fileio import BlobReader
client = Client("apache-beam-testing")
bucket = Bucket(client=client, name="apache-beam-samples")
test_files = [
"textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz",
"textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz",
"textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz",
"textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz",
"textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz",
"textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz",
"textio/textio-test-data.default.1k.txt",
"textio/textio-test-data.default.1k.txt.gz",
"textio/textio-test-data.gzip-local.1k.txt.gz",
]
for raw_download in [False, True]:
print(f"-- raw_download={raw_download} --")
for test_file in test_files:
blob = Blob(test_file, bucket=bucket)
reader = BlobReader(blob, raw_download=raw_download)
a = reader.read(16777216)
b = reader.read(16777216)
print(f"[{test_file}] first_read:{len(a)}, second_read:{len(b)}")
Reproduction steps: supporting files
Run the code directly.
Reproduction steps: actual results
-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:101000 <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:76478 <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0
Reproduction steps: expected results
-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0
OS & version + platform
MacOS, Linux
Python environment
Python 3.12
Python dependencies
google-cloud-storage 3.14.1
Additional context
No response
- Lingua principale
- Python
- Stelle
- 5.4k
- Fork
- 1.8k
- Merge medio
- 2g 20h
- PR unite (30g)
- 140
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di googleapis/google-cloud-python
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
googleapis/google-cloud-python#19099 ·
I maintainer di solito rispondono entro 2 giorni
-
firestore: field path with a quoted name ending in a backslash is misparsedForse già presa @sebxt l’ha presa 28 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
googleapis/google-cloud-python#18771 ·
I maintainer di solito rispondono entro 2 giorni
-
api: spanner type: feature request
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
googleapis/google-cloud-python#18584 ·
I maintainer di solito rispondono entro 2 giorni
-
auth: `impersonated_credentials` loses `subject` on copy, so `with_scopes()` and `with_quota_project()` turn off domain wide delegationForse già presa @Om-singhaI l’ha presa 57 giorni fa. Apertaauth priority: p2 type: bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
googleapis/google-cloud-python#18428 ·
I maintainer di solito rispondono entro 2 giorni
-
A client called `run_in_transaction` be stuck if Spanner is downForse già presa @founderinaction l’ha presa 10 giorni fa. Apertapriority: p2 type: bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
googleapis/google-cloud-python#18375 · 1 commento ·
I maintainer di solito rispondono entro 2 giorni
Tutte le issue di googleapis/google-cloud-python
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 64/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
btclib-org/btclib-wallet#1045 ·
I maintainer di solito rispondono entro 1 giorno
-
area-templates triage:bot-seen triage:needs-human
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
microsoft/aspire#22356 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
I maintainer di solito rispondono entro 1 giorno
-
[Train] An elastic ScalingConfig(num_workers=(0, 0)) is accepted and the run waits foreverForse già presa @wguo7 l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
ray-project/ray#68217 ·
I maintainer di solito rispondono entro 2 giorni