Apply Dask to parallelise acs_regional_stats
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- jupyter-notebook
- Domain
- data, performance
Research direction
Start with example_notebooks/FAQ_example_timeseries_stats.ipynb and the acs_regional_stats entry point, then run the current multi-year daily calculation to establish its memory use and runtime. Develop the requested Dask-based example that calculates area averages per file, and verify that it produces regional time series while reducing the current several-minute, memory-intensive workload.
Written by the indexing model from the issue text.
Description
acs_regional_stats can be very memory intensive to run, particularly over many regions and many timesteps.
We should develop an example of running acs_regional_stats for many years of daily data to produce area averaged timeseries for regions. Currently, this is possible, but will take several minutes to calculate.
Dask is likely to be able to achieve this by calculating area averages per file.
Previous development has focused on reducing memory usage through other clever means, such as implementing chunks to reduce the number of timesteps loaded into the memory to calculate stats over each time. This could be parallelised, but it is not currently.
- Dominant language
- Jupyter Notebook
- Stars
- 8
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from AusClimateService/plotting_maps
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Improve plot_titles Open
AusClimateService/plotting_maps#37 · 2 comments · 1 assignee ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
-
Update cmap_dict Open
AusClimateService/plotting_maps#33 · 1 assignee ·
All issues in AusClimateService/plotting_maps
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
channels:add check:passed
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
avniproject/avni-webapp#1811 ·
-
inceleme-kuyrugu
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Greater-Turkiye/platform#114 ·
-
bug triage
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
ultralytics/ultralytics#26305 · 1 comment ·