google-cloud-storage: Failed to handle doubly-gzipped files

オープン
#18,423 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

  • #18430 @ohmayr による — マージされずにクローズ

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
68/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
google-cloud, python
領域
cloud

調査の方向性

Start by running the main.py reproduction and tracing the BlobReader read path, especially how it handles content and transfer encoding for the doubly-gzipped object. Add regression coverage for both raw_download values and verify that the second read returns 0, matching the expected results.

索引モデルが issue の本文から書いたものです。

説明

api: storage priority: p3 type: bug
Determine this is the right repository
  • I determined this is the correct repository in which to report this bug.
Summary of the issue

When handling a doubly gzipped file (the first test file in the code below), the reader fails to reach the end of file.

Image
API client name and version

google-cloud-storage 3.14.1

Reproduction steps: code

file: main.py

from google.cloud.storage import Blob
from google.cloud.storage import Bucket
from google.cloud.storage import Client

from google.cloud.storage.fileio import BlobReader

client = Client("apache-beam-testing")
bucket = Bucket(client=client, name="apache-beam-samples")

test_files = [
    "textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz",
    "textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz",
    "textio/textio-test-data.default.1k.txt",
    "textio/textio-test-data.default.1k.txt.gz",
    "textio/textio-test-data.gzip-local.1k.txt.gz",
]

for raw_download in [False, True]:
  print(f"-- raw_download={raw_download} --")
  for test_file in test_files:
    blob = Blob(test_file, bucket=bucket)
    reader = BlobReader(blob, raw_download=raw_download)
    a = reader.read(16777216)
    b = reader.read(16777216)
    print(f"[{test_file}] first_read:{len(a)}, second_read:{len(b)}")
Reproduction steps: supporting files

Run the code directly.

Reproduction steps: actual results
-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:101000   <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:76478   <--- problem
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0
Reproduction steps: expected results
-- raw_download=False --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:101000, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:101000, second_read:0
-- raw_download=True --
[textio/textio-test-data.content-type-gzip-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-gzip-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-none-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-gzip.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.content-type-text-content-encoding-none.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.default.1k.txt] first_read:101000, second_read:0
[textio/textio-test-data.default.1k.txt.gz] first_read:76478, second_read:0
[textio/textio-test-data.gzip-local.1k.txt.gz] first_read:76484, second_read:0
OS & version + platform

MacOS, Linux

Python environment

Python 3.12

Python dependencies

google-cloud-storage 3.14.1

Additional context

No response

主要言語
Python
スター
5.4k
フォーク
1.8k
平均マージ
2日 20時間
マージ済み PR(30日)
140

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

googleapis/google-cloud-python のほかの issue

googleapis/google-cloud-python の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。