Annual NetCDF merge writes values to the wrong ensemble members and timesteps
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 80/100
Research direction
Start with base/utils/R/combine_model_netcdf.R around the per-site writer at lines 153–183, and inspect the existing tests and the ncvar_put() dimension ordering. Reproduce the issue with the bundled ensemble fixtures using the R code in the report, then add a regression test comparing both members’ merged AGB values across timesteps with their source values. Done means the two-member merge preserves those values and the single-member control still matches.
Written by the indexing model from the issue text.
Description
Bug Description
When I merge two ensemble members with nc_merge_all_sites_by_year(), I expect each aboveground biomass (AGB) value to stay with its original site, member and timestep, but I observe 5,809 of 5,840 values at site 101 differ from their source values. The merge completes successfully and the output dimensions are correct.
To Reproduce
- From the PEcAn repository root at
368c3b76086e2891cbd12fcebd9b3e422fb9d579, usePEcAn.utilsfrom that checkout, withncdf4anddoSNOWinstalled andcdoon your PATH. - Run the R code below. It copies the bundled 2019 NetCDF fixtures into a temporary directory under numeric site IDs 101 and 102. It runs the annual merge with one member as a control, then with both members.
- Compare the merged
AGBseries at site 101 with the original series for each member.
Runnable R reproduction
stopifnot(requireNamespace("PEcAn.utils"), requireNamespace("ncdf4"))
fixtures <- "base/utils/tests/testthat/data/ensemble_fixtures"
work <- tempfile()
dir.create(work)
read_var <- function(path, name) {
f <- ncdf4::nc_open(path)
on.exit(ncdf4::nc_close(f))
ncdf4::ncvar_get(f, name)
}
series <- lapply(1:2, function(i) {
dir <- file.path(work, sprintf("ENS-%05d-101", i))
dir.create(dir)
src <- file.path(fixtures, sprintf("ENS-%05d-e968e9c8f8574cb2/2019.nc", i))
stopifnot(file.copy(src, dir))
other <- file.path(work, sprintf("ENS-%05d-102", i))
dir.create(other)
stopifnot(file.copy(file.path(fixtures, sprintf("ENS-%05d-ebb783e86d2ac6fb/2019.nc", i)), other))
read_var(src, "AGB")
})
for (n in 1:2) {
dest <- file.path(work, paste0("merged", n))
dir.create(dest)
path <- PEcAn.utils::nc_merge_all_sites_by_year(
work, dest, n, c(101, 102), "2019-01-01", "2019-12-31", cores = 1
)
h <- ncdf4::nc_open(path)
dims <- vapply(h$var$AGB$dim, function(x) x$name, "")
stopifnot(identical(dims, c("site", "ensemble", "time")))
at <- match(101, ncdf4::ncvar_get(h, "site_id"))
block <- ncdf4::ncvar_get(h, "AGB", start = c(at, 1, 1),
count = c(1, n, -1), collapse_degen = FALSE)
ncdf4::nc_close(h)
actual <- matrix(block[1,,,drop=FALSE], nrow = n)
expected <- do.call(rbind, series[seq_len(n)])
stopifnot(identical(dim(actual), dim(expected)))
cat(sprintf("%d member(s): %d of %d AGB cells differ\n", n,
sum(actual != expected), length(expected)))
if (n == 2) {
wrong <- which(actual != expected, arr.ind=TRUE)[1, ]
print(c(member=wrong[1], timestep=wrong[2], observed=actual[wrong[1],wrong[2]],
expected=expected[wrong[1],wrong[2]]))
}
}
Expected behavior
At site 101, member 2, timestep 1, the merged file should contain the source value 1.807129. It contains 1.549805 (values shown to six decimal places).
| Members merged | Expected mismatches at site 101 | Actual mismatches |
|---|---|---|
| 1 | 0 | 0 of 2,920 |
| 2 | 0 | 5,809 of 5,840 |
Screenshots
N/A. The reproduction compares values directly from the NetCDF files.
Machine
- Server: local workstation
- OS: Fedora Linux 44
- Browser: N/A
- Version: PEcAn development commit
368c3b7608, R 4.6.2, CDO 2.6.5
Additional context
The values already differ in the per-site file before CDO combines the sites. The per-site writer builds a matrix with timesteps in rows and members in columns. The destination variable has dimensions site × ensemble × time, so for a single site the write needs members to vary first. The sequential data assimilation workflow calls this merge.
I’d like to work on this issue. My plan is to add a regression test that compares both members’ merged values with their source values across multiple timesteps, then transpose the per-site matrix before passing it to ncvar_put(). Could you assign this to me?
- Dominant language
- R
- Stars
- 246
- Forks
- 322
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 15
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PecanProject/pecan
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
PecanProject/pecan#4255 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
PecanProject/pecan#4254 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
PecanProject/pecan#4253 · 2 comments ·
Maintainers usually reply within 1 day
-
data.land: prepare_pools fails to calculate woody biomass from TotLivBiom when AbvGrndWood is present without coarse rootsPossibly taken @coder-Yash886 claimed this 4 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 82/100
PecanProject/pecan#4243 · 3 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
PecanProject/pecan#4236 ·
Maintainers usually reply within 1 day
All issues in PecanProject/pecan
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
OHDSI/PhenotypeR#728 ·
-
DataSet inherits from DotNetWrapper but has no .NET pointer, so NetObject methods failPossibly taken @PavelBal claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Open-Systems-Pharmacology/OSPSuite-R#2076 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
rfordatascience/tidytuesday#1105 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Huber-group-EMBL/remart#17 · 1 comment ·
-
teaching-followup
Difficulty 1/5 Under an hour Newbie friendliness 75/100
Openscapes/kyber#197 ·