HDF dataset strides storage (pot. enhancement?)

Open
  • Is something wrong?
#52 10 comments 0 reactions 1 assignee View on GitHub

@gheber is already working on this.

Since Apr 18, 2016.

Assessment

This issue has not been assessed yet.

Description

documentation question

@gheber Should I post this to [HDF-forum]?

The original element datatype of a dataset is stored by HDF5 and is required to identify the way to actually read back the elements into memory. The HDF API allows to provide the dataset values as arbitrary types. Internally the conversion is done transparently to the user. Also, the dimension lengths are stored as obligatory dataset attribute.

In order to be compatible to arbitrary applications, next to the dimension lengths and the element type, a 3rd information would be necessary to store and manage all properties of the array (let's call the dataset this way here): the strides of the dimensions. This would enable HDF5 to handle row-major vers. column-major order, and also any other custom stride the dataset storage might possibly be stored in.

The HDF5 dataset documentation (cannot find the exact place right now) points out the difference in storage scheme when using different languages. Datasets stored in ILNumerics / FORTRAN will have the dimensions flipped when read in C and vice versa.

Currently, I see the following options to handle this situation:

  • The user of HDF5 needs to keep the actual dimensions in mind. Dimensions need to be flipped manually when reading data which were stored using the other convention.
  • The hyperslab feature seems to be flexible enough to be used to read from the dataset storage utilizing arbitrary strides (?). Drawback: the flexibility comes to the price of a performance drop compared to sequential memory reads.

HDF5 stores the dataset in _row major _order, no matter what. Currently, a user who wants the data to be readable using any storage convention, could / must use an attribute in order to attach this information to the dataset. The necessary dimension permutations must be implemented on a layer outside of HDF5 still. There is no standard place / attribute name which is recommended for this (?).

I am interested in your experiences regarding this issue and in your recommendation on how to handle different storage schemes. Should we consider HDF5 as 'being row-major' only? Or is it rather meant to be a general storage API, capable of handling any storage scheme?

Maybe there are already plans to incorporate this extra bit into HDF? Having the information of the storage scheme of the dataset available would help the user of HDF5 to identify the necessary actions to take in order to clean up any dimension order mess, at least. The attribute option is a reasonable fallback. But an official 'standard' place to store / read the strides could help compatibility significantly. I am not aware of all potential side effects in HDF5 but to define a standard attribute might be possible without introducing too huge problems? [For an optimal solution, HDF would auto-permute the dimensions, but this would obviously be more demanding.]

What do you think?

Dominant language
C#
Stars
87
Forks
32
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HDFGroup/HDF.PInvoke

All issues in HDFGroup/HDF.PInvoke

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.