GzipReader's encoding conversion behaves differently from IO
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
Research direction
This reports encoding-conversion mismatches between Zlib::GzipReader and IO, with three reproducible examples in the issue body. Start by finding where GzipReader applies external_encoding/internal_encoding in the C extension (ext/zlib/zlib.c) and compare with how io.c handles conversion errors and the gets limit. Note the maintainer is unsure whether fixing is desirable due to compatibility concerns, so the first step is discussing intended behavior before any patch; 'done' is a maintainer decision plus, if agreed, matching IO semantics with tests covering the three cases.
Written by the indexing model from the issue text.
Description
When external_encoding/internal_encoding are given, Zlib::GzipReader converts the data differently from IO in a few ways. Fixing them could be incompatible, so I'm not sure whether they are worth fixing. I'm opening this issue for now.
-
A conversion error is ignored, and the data is returned as is.
# "a" in UTF-16BE followed by one extra byte gz = Zlib.gzip("\x00a\x00".b) o = {external_encoding: "UTF-16BE", internal_encoding: "UTF-8"} Zlib::GzipReader.new(StringIO.new(gz), **o).read #=> "\x00a\x00" (not converted) # IO#read raises Encoding::InvalidByteSequenceErrorWith a dummy encoding such as UTF-16 (with BOM),
getcreturns""and the extra byte is lost:gz = Zlib.gzip("\xFE\xFF\x00a\x00".b) o = {external_encoding: "UTF-16", internal_encoding: "UTF-8"} g = Zlib::GzipReader.new(StringIO.new(gz), **o) 3.times.map { g.getc } #=> ["a", "", nil] -
The limit of
getsis counted in bytes before conversion, while IO counts it after conversion.s = ("\x82\xa0" * 3 + "\n").b # "あああ\n" in Shift_JIS o = {external_encoding: "Shift_JIS", internal_encoding: "UTF-8"} Zlib::GzipReader.new(StringIO.new(Zlib.gzip(s)), **o).gets(3) #=> "ああ" # File#gets(3) with the same options returns "あ"
- Dominant language
- C
- Stars
- 73
- Forks
- 43
- Avg merge
- 4d 18m
- Merged PRs (30d)
- 12
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from ruby/zlib
-
`Zlib::Deflate#params` segfaults when called before any output buffer existsPossibly taken @SAY-5 claimed this 85 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 72/100
Maintainers usually reply within 2 days
-
Difficulty 3/5 1-2 days Newbie friendliness 55/100
Maintainers usually reply within 2 days
-
GzipReader: gets sometimes returns incomplete multi-byte charactersPossibly taken @segiddins claimed this 22 days ago. Open
Difficulty 3/5 1-2 days Newbie friendliness 42/100
Maintainers usually reply within 2 days
-
GzipReader: internal_encoding option overwrites external_encodingPossibly taken @segiddins claimed this 22 days ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Maintainers usually reply within 2 days
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 2 days
Similar issues
-
HTTP Request Smuggling: non-final chunked Transfer-Encoding not rejected (CWE-444)Possibly taken @code1a2024 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
jaredgei/wow-forever-android#97 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
openziti/ziti-sdk-c#1254 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
flatpak/xdg-desktop-portal#2215 ·
Maintainers usually reply within 2 days
-
http_fetch tests: the two no-child rows fail, not skip, where a task's children file does not existPossibly taken A pull request linked to this issue is open or already merged. Openarea:http kind:gate-defect
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
InauguralSystems/EigenScript#2397 ·
Maintainers usually reply within 1 day