Feature Request: Persistent file storage from Hermes
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- cpp
- Domain
- data-engineering
Research direction
The issue names Hermes, ADIOS2, and the bp5 format but does not identify source files, tests, or an entry point. Start by locating the data-clearing and shutdown paths, then clarify the persistence option, output lifecycle, and acceptance tests needed to show that collected data is available in bp5 format after the workflow ends.
Written by the indexing model from the issue text.
Description
Requested feature:
Persistent file storage from Hermes
Problem:
We are using Hermes as the IO engine for ADIOS2 in scientific workflows.
Whenever our workflow finishes, Hermes has no means to store the collected data to disk and therefore the data is lost.
As the raw data from the workflow is not saved it is not possible to perform post-hoc analysis.
As these workflows can be computationally expensive it is not practical to re-run experiments to regenerate the data.
Further, if the workflow is of a stochastic or chaotic nature, it may not be possible to replicate previous runs.
Proposed solution:
When Hermes is preparing to clear data, add an option so that this data can be written to disk.
For example, the default engine for ADIOS2 is the bp5 engine.
Hermes could be configured to write to disk data that it no longer is using with the bp5 format.
At the end of the workflow when Hermes exits, an end user would have access to the raw data in bp5 format.
This way the user gets the benefit of using the Hermes engine while in the workflow but also can access the results of the experiment after it was run.
- Dominant language
- C++
- Stars
- 34
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HDFGroup/hermes
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Figure is missing on Wiki pageMay be free again @lukemartinlogan claimed this 785 days ago, and no pull request is open. Opendocumentation
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
Similar issues
-
documentation
Difficulty 1/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
grumpycoders/pcsx-redux#2667 ·
-
GCC 15.2: -Wcast-qual warning - cast discards const qualifier from pointer target type (H5trace.c)Open
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
google/tensorstore#305 ·
-
bug module: copilot
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
MaaAssistantArknights/MaaAssistantArknights#19250 · 2 comments ·
Maintainers usually reply within 1 day