`Zlib::GzipReader` doesn't read some large files

Open
#50 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
ruby
Domain
tooling

Research direction

Start by reproducing the report with Zlib::GzipReader.open on adresses-france.csv.gz and compare its output with the gzip command-line result. Check whether the behavior is specific to the compressed file or to reading large files; done means identifying why the complete decompressed content is not returned.

Written by the indexing model from the issue text.

Description

Hi.
I have to inflate a .csv.gz file which should return a 4 GB CSV with 25 million rows.

When I use an app or the gzip command line, I get the full file without issue.
When I use Zlib::GzipReader, only the first row is returned.

> Zlib::GzipReader.open("adresses-france.csv.gz") { |gz|  print gz.read }
id;id_fantoir;numero;rep;nom_voie;code_postal;code_insee;nom_commune;code_insee_ancienne_commune;nom_ancienne_commune;x;y;lon;lat;type_position;alias;nom_ld;libelle_acheminement;nom_afnor;source_position;source_nom_voie;certification_commune;cad_parcelles
 => nil

The file is provided by the french government:

There are many other files in the directory (for each region) but I cannot reproduce the issue with other files.

This service also provided a similar file in Addok format (https://adresse.data.gouv.fr/data/ban/adresses/latest/addok/adresses-addok-france.ndjson.gz) which should return a 3GB file with 2 million rows, but only the 25k first rows are returned by Zlib::GzipReader.

Is there any limit to what Zlib can support ? (size, rows, ..)
Does it come from the compressed file ?

Dominant language
C
Stars
73
Forks
43
Avg merge
4d 18m
Merged PRs (30d)
12

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ruby/zlib

All issues in ruby/zlib

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.