openPMD/openPMD-api

Streaming: Histogramming of Particle Data

Open

#1,248 opened on Apr 19, 2022

View on GitHubContributor guide
 (0 comments) (3 reactions) (1 assignee)C++ (57 forks)auto 404
help wantedtools

Repository metrics

Stars
 (160 stars)
PR merge metrics
 (PR metrics pending)

Description

@aeriforme would like to write a data pipeline that takes all particle data and bins it to a 1D or 2D user-defined histogram. This would be a great application of our work in @franzpoeschel et al. https://arxiv.org/abs/2107.06108.

Idea: Let's take the script in https://github.com/openPMD/openPMD-api/blob/dev/src/cli/pipe.py as a template and instead of forwarding the data 1:1 to another file format, we call np.histogram (1D) or np.histogram2d (2D) to store into a histogram per process. Finally, we reduce the histograms (sum) via mpi4py and write them out into a 2D mesh (or simple np.save from rank 0).

@aeriforme works with CINECA clusters.

Ref.:

Contributor guide

Research direction
Review the template script `src/cli/pipe.py` for streaming particle data via openPMD. Explore the openPMD Python API to access particle records. Then apply `numpy.histogram` or `numpy.histogram2d` to bin data and use `mpi4py` to sum histograms across MPI ranks.
Tech stack
pythonnumpy
Domain
stream processingdata
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Pythonnumpympi4py
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50