openPMD/openPMD-api

Series Open Performance

Open

#485 opened on Feb 19, 2019

View on GitHubContributor guide
 (9 comments) (0 reactions) (0 assignees)C++ (57 forks)auto 404
good first issuehelp wantedinternal

Repository metrics

Stars
 (160 stars)
PR merge metrics
 (PR metrics pending)

Description

We should investigate and profile our performance of Series open (especially with READ_ONLY and READ_WRITE, but also with CREATE on truncation: #483).

Most crucial with file-based series, I have the subjective feeling they are slower than they need to be, already in serial mode.

We might need to clearly define also in this step, what we check and assume when opening a series (aka contract). The less assumptions the better, if you ask me. For example, I would not expect the data of each iteration nor it's names or layouts to be required to match. (Although some applications, such as our openPMD-viewer GUI, will benefit if we later on provide an auxiliary scan-like list summary for records in all iterations.)

Resources

Maybe some useful resources:

Contributor guide

Research direction
Investigate the Series::open code path for file based series. Use profiling tools like strace and perf to identify bottlenecks in READ ONLY, READ WRITE, and CREATE modes. Compare the current assumptions about iteration layouts and consider whether they can be relaxed. Look at the existing filesystem operations and the auxiliary::Filesystem class. Check issue #483 about truncation and #462 about time logs. Start by writing a micro benchmark to measure open times.
Tech stack
cpp
Domain
backend
Issue type
Performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
C++GitProfiling tools
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35