Variable length array

Open
  • Is something wrong?
#85 2 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@mattjala is already working on this.

Since Sep 17, 2026.

Assessment

This issue has not been assessed yet.

Description

enhancement

I'm creating a backend to use transparently pytables, h5py and h5pyd. I ran my test suite on h5pyd and am confronted with the issue of the variable length array. In h5py, they manage to do it using a "special" dtype: h5py.string_dtype or h5py.vlen_dtype . After some digging in I found in h5pyd, the function special_dtype where the docstring seems promising:

    vlen = basetype
        Base type for HDF5 variable-length datatype. This can be Python
        str type or instance of np.dtype.
        Example: special_dtype( vlen=str )

however after trying it out it seems its working only in the case where vlen=str and not any numpy dtype. Using a special dtype of np.uint32s I could create a dataset but when trying to access a given element I got this traceback:
File "<ipython-input-114-66578725db5f>", line 1, in <module> dset[0] File "C:\Miniconda3\envs\pymodaq_dev\lib\site-packages\h5pyd\_hl\dataset.py", line 802, in __getitem__ arr1d = bytesToArray(rsp, mtype, page_mshape) File "C:\Miniconda3\envs\pymodaq_dev\lib\site-packages\h5pyd\_hl\base.py", line 503, in bytesToArray offset = readElement(data, offset, arr, index, dt) File "C:\Miniconda3\envs\pymodaq_dev\lib\site-packages\h5pyd\_hl\base.py", line 467, in readElement arr[index] = vlen(0) TypeError: 'numpy.dtype' object is not callable

Then a bit further in the code I found :

def check_dtype(**kwds):
""" Check a dtype for h5py special type "hint" information. Only one
keyword may be given.

vlen = dtype
    If the dtype represents an HDF5 vlen, returns the Python base class.
    Currently only builting string vlens (str) are supported.  Returns
    None if the dtype does not represent an HDF5 vlen.

So the question is: is it or will it be possible to use any numpy dtype for variable length arrays in h5pyd?

Thx

Dominant language
Python
Stars
124
Forks
48
Avg merge
6d 2h
Merged PRs (30d)
16

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HDFGroup/h5pyd

All issues in HDFGroup/h5pyd

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.