Annual NetCDF merge writes values to the wrong ensemble members and timesteps

Open Beginner friendly
#4,251 3 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@ANAMASGARD is already working on this.

Since Nov 17, 2026.

  • #4252 by @ANAMASGARD — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
80/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
r
Domain
data

Research direction

Start with base/utils/R/combine_model_netcdf.R around the per-site writer at lines 153–183, and inspect the existing tests and the ncvar_put() dimension ordering. Reproduce the issue with the bundled ensemble fixtures using the R code in the report, then add a regression test comparing both members’ merged AGB values across timesteps with their source values. Done means the two-member merge preserves those values and the single-member control still matches.

Written by the indexing model from the issue text.

Description

Bug Description

When I merge two ensemble members with nc_merge_all_sites_by_year(), I expect each aboveground biomass (AGB) value to stay with its original site, member and timestep, but I observe 5,809 of 5,840 values at site 101 differ from their source values. The merge completes successfully and the output dimensions are correct.

To Reproduce
  1. From the PEcAn repository root at 368c3b76086e2891cbd12fcebd9b3e422fb9d579, use PEcAn.utils from that checkout, with ncdf4 and doSNOW installed and cdo on your PATH.
  2. Run the R code below. It copies the bundled 2019 NetCDF fixtures into a temporary directory under numeric site IDs 101 and 102. It runs the annual merge with one member as a control, then with both members.
  3. Compare the merged AGB series at site 101 with the original series for each member.
Runnable R reproduction
stopifnot(requireNamespace("PEcAn.utils"), requireNamespace("ncdf4"))
fixtures <- "base/utils/tests/testthat/data/ensemble_fixtures"
work <- tempfile()
dir.create(work)
read_var <- function(path, name) {
  f <- ncdf4::nc_open(path)
  on.exit(ncdf4::nc_close(f))
  ncdf4::ncvar_get(f, name)
}
series <- lapply(1:2, function(i) {
  dir <- file.path(work, sprintf("ENS-%05d-101", i))
  dir.create(dir)
  src <- file.path(fixtures, sprintf("ENS-%05d-e968e9c8f8574cb2/2019.nc", i))
  stopifnot(file.copy(src, dir))
  other <- file.path(work, sprintf("ENS-%05d-102", i))
  dir.create(other)
  stopifnot(file.copy(file.path(fixtures, sprintf("ENS-%05d-ebb783e86d2ac6fb/2019.nc", i)), other))
  read_var(src, "AGB")
})
for (n in 1:2) {
  dest <- file.path(work, paste0("merged", n))
  dir.create(dest)
  path <- PEcAn.utils::nc_merge_all_sites_by_year(
    work, dest, n, c(101, 102), "2019-01-01", "2019-12-31", cores = 1
  )
  h <- ncdf4::nc_open(path)
  dims <- vapply(h$var$AGB$dim, function(x) x$name, "")
  stopifnot(identical(dims, c("site", "ensemble", "time")))
  at <- match(101, ncdf4::ncvar_get(h, "site_id"))
  block <- ncdf4::ncvar_get(h, "AGB", start = c(at, 1, 1),
                            count = c(1, n, -1), collapse_degen = FALSE)
  ncdf4::nc_close(h)
  actual <- matrix(block[1,,,drop=FALSE], nrow = n)
  expected <- do.call(rbind, series[seq_len(n)])
  stopifnot(identical(dim(actual), dim(expected)))
  cat(sprintf("%d member(s): %d of %d AGB cells differ\n", n,
              sum(actual != expected), length(expected)))
  if (n == 2) {
    wrong <- which(actual != expected, arr.ind=TRUE)[1, ]
    print(c(member=wrong[1], timestep=wrong[2], observed=actual[wrong[1],wrong[2]],
            expected=expected[wrong[1],wrong[2]]))
  }
}

Expected behavior

At site 101, member 2, timestep 1, the merged file should contain the source value 1.807129. It contains 1.549805 (values shown to six decimal places).

Members merged Expected mismatches at site 101 Actual mismatches
1 0 0 of 2,920
2 0 5,809 of 5,840

Screenshots

N/A. The reproduction compares values directly from the NetCDF files.

Machine

  • Server: local workstation
  • OS: Fedora Linux 44
  • Browser: N/A
  • Version: PEcAn development commit 368c3b7608, R 4.6.2, CDO 2.6.5

Additional context

The values already differ in the per-site file before CDO combines the sites. The per-site writer builds a matrix with timesteps in rows and members in columns. The destination variable has dimensions site × ensemble × time, so for a single site the write needs members to vary first. The sequential data assimilation workflow calls this merge.

I’d like to work on this issue. My plan is to add a regression test that compares both members’ merged values with their source values across multiple timesteps, then transpose the per-site matrix before passing it to ncvar_put(). Could you assign this to me?

Dominant language
R
Stars
246
Forks
322
Avg merge
4d 15h
Merged PRs (30d)
15

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PecanProject/pecan

All issues in PecanProject/pecan

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.