How to handle snappy files generated by Trino?
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
Research direction
Start with the provided reproduction and the traceback in snappy/snappy_formats.py, especially get_decompress_function and guess_format_by_header. Compare behavior between versions 0.7.0 and 0.7.1 using the Trino-generated file. Done means the file format is detected and decompressed successfully without regressing the earlier output.
Written by the indexing model from the issue text.
Description
Hello,
With the new release to 0.7.1 the I can't decompress CSV files generated by Trino, I think the issue is related with the Hadoop_snappy. Does anyone know how it can fixed?
from snappy import snappy_formats
csv_file = 'csv_67dba65a.snappy'
def read_file(file_path):
return open(file_path, 'rb')
decompress_func, read_chunk = snappy_formats.get_decompress_function(
'auto',
read_file(csv_file)
)
decompressed_stream = io.BytesIO()
# Decompress the data
decompress_func(
read_file(csv_file),
decompressed_stream,
start_chunk=read_chunk
)
decompressed_stream.seek(0)
print(f"Compressed file: {read_file(csv_file).read()}")
print(f"DeCompressed file: {decompressed_stream.read()}")
This code has different outputs based on the version:
-
0.7.0
Compressed file: b'\x00\x00\x00\x04\x00\x00\x00\x06\x04\x0c"a"\n'
DeCompressed file: b'"a"\n"a"\n' -
0.7.1
.venv/lib/python3.12/site-packages/snappy/snappy_formats.py", line 64, in get_decompress_function
decompress_func, read_chunk = guess_format_by_header(fin)
.venv/lib/python3.12/site-packages/snappy/snappy_formats.py", line 59, in guess_format_by_header
raise UncompressError("Can't detect archive format")
snappy.snappy.UncompressError: Can't detect archive format
- Dominant language
- Python
- Stars
- 490
- Forks
- 104
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intake/python-snappy
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
intake/python-snappy#152 · 3 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
intake/python-snappy#151 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
intake/python-snappy#150 · 2 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
intake/python-snappy#147 · 3 comments ·
-
Difficulty 3/5 1-2 days Newbie friendliness 25/100
intake/python-snappy#138 · 2 comments ·
All issues in intake/python-snappy
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100