Serving datasets with bitshuffle compression in POSIX files

Open
  • Is something wrong?
#76 34 comments 0 reactions 1 assignee View on GitHub

@mattjala is already working on this.

Since Aug 25, 2026.

Assessment

This issue has not been assessed yet.

Description

enhancement

I am trying to serve POSIX files that contain datasets compressed with the bitshuffle filter.

hsload --link works without any trouble as do requests to metadata and uncompressed datasets. However, requests to the compressed datasets fail with the following errors in the datanode:

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.8/site-packages/hsds/chunk_dn.py", line 259, in GET_Chunk
    chunk_arr = await get_chunk(app, chunk_id, dset_json, bucket=bucket, s3path=s3path, s3offset=s3offset, s3size=s3size, chunk_init=False)
  File "/usr/local/lib/python3.8/site-packages/hsds/datanode_lib.py", line 386, in get_chunk
    chunk_arr = bytesToArray(chunk_bytes, dt, dims)
  File "/usr/local/lib/python3.8/site-packages/hsds/util/arrayUtil.py", line 453, in bytesToArray
    arr = np.frombuffer(data, dtype=dt)
ValueError: buffer size must be a multiple of element size

and

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.8/site-packages/hsds/chunk_dn.py", line 259, in GET_Chunk
    chunk_arr = await get_chunk(app, chunk_id, dset_json, bucket=bucket, s3path=s3path, s3offset=s3offset, s3size=s3size, chunk_init=False)
  File "/usr/local/lib/python3.8/site-packages/hsds/datanode_lib.py", line 386, in get_chunk
    chunk_arr = bytesToArray(chunk_bytes, dt, dims)
  File "/usr/local/lib/python3.8/site-packages/hsds/util/arrayUtil.py", line 459, in bytesToArray
    arr = arr.reshape(shape)
ValueError: cannot reshape array of size 6987 into shape (66,123)

.

How should I proceed to be able to request such datasets ? Given that I do hsload --link, should I look into HSDS rather than h5pyd ?

Dominant language
Python
Stars
158
Forks
61
Avg merge
4d 17h
Merged PRs (30d)
6

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HDFGroup/hsds

All issues in HDFGroup/hsds

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.