[HELP] Wiki and demo code about read ultra large dataset

Open
  • Is something wrong?
#186 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Documentation
Clarity
Needs clarification
Activity status
Stale
Tech stack
csharp
Domain
documentation

Research direction

Start with the Wiki's “Cookbook: Datasets” page and the existing demo code for datasets. Determine how HDF.PInvoke can read data larger than 2GB in chunks without exceeding the .NET array limit, then document the approach and add a working ultra-large-dataset example.

Written by the indexing model from the issue text.

Description

Hi,

Many thanks for this great data format library.

I'm working on read a ultra large hdf5 dataset via HDF.PInvoke recently. Due to the reason of .NET array can not be allocated in size greater than 2GB, so if the data inside the hdf5 file its size is greater than 2GB, then the program crashed.

The dataset which its size less than 2GB could be read via the demo code in wiki Cookbook : Datasets, and I guess i could read the dataset which its size greater than 2GB chunk by chunk via HDF.PInvoke, but after google search around it, i was not able to figure out how to implements such code correctly.

Here is the DEMO data in ultra large size that i currently working on: download the P_1900 | Human Prostate dataset from this repository.

Dominant language
C#
Stars
87
Forks
32
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HDFGroup/HDF.PInvoke

All issues in HDFGroup/HDF.PInvoke

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.