GzipReader's encoding conversion behaves differently from IO

Open
#166 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
c, ruby
Domain
backend

Research direction

This reports encoding-conversion mismatches between Zlib::GzipReader and IO, with three reproducible examples in the issue body. Start by finding where GzipReader applies external_encoding/internal_encoding in the C extension (ext/zlib/zlib.c) and compare with how io.c handles conversion errors and the gets limit. Note the maintainer is unsure whether fixing is desirable due to compatibility concerns, so the first step is discussing intended behavior before any patch; 'done' is a maintainer decision plus, if agreed, matching IO semantics with tests covering the three cases.

Written by the indexing model from the issue text.

Description

When external_encoding/internal_encoding are given, Zlib::GzipReader converts the data differently from IO in a few ways. Fixing them could be incompatible, so I'm not sure whether they are worth fixing. I'm opening this issue for now.

  1. A conversion error is ignored, and the data is returned as is.

    # "a" in UTF-16BE followed by one extra byte
    gz = Zlib.gzip("\x00a\x00".b)
    o = {external_encoding: "UTF-16BE", internal_encoding: "UTF-8"}
    Zlib::GzipReader.new(StringIO.new(gz), **o).read
    #=> "\x00a\x00" (not converted)
    # IO#read raises Encoding::InvalidByteSequenceError
    

    With a dummy encoding such as UTF-16 (with BOM), getc returns "" and the extra byte is lost:

    gz = Zlib.gzip("\xFE\xFF\x00a\x00".b)
    o = {external_encoding: "UTF-16", internal_encoding: "UTF-8"}
    g = Zlib::GzipReader.new(StringIO.new(gz), **o)
    3.times.map { g.getc } #=> ["a", "", nil]
    
  2. The limit of gets is counted in bytes before conversion, while IO counts it after conversion.

    s = ("\x82\xa0" * 3 + "\n").b # "あああ\n" in Shift_JIS
    o = {external_encoding: "Shift_JIS", internal_encoding: "UTF-8"}
    Zlib::GzipReader.new(StringIO.new(Zlib.gzip(s)), **o).gets(3) #=> "ああ"
    # File#gets(3) with the same options returns "あ"
    
Dominant language
C
Stars
73
Forks
43
Avg merge
4d 18m
Merged PRs (30d)
12

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ruby/zlib

All issues in ruby/zlib

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.