Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

NSE and R2 return 1.0 when model values are NaN

Open Beginner friendly
  • Is something wrong?
#720 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

  • #721 by @syahra712 — closed without merging

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
numpy, pandas, python
Domain
analytics

Research direction

Start in src/modelskill/metrics.py, in nash_sutcliffe_efficiency, where np.sum is applied to the inputs; r2 builds on the same path. Reproduce with the pandas Series example from the issue, comparing against the NumPy-array call, which already returns NaN. Done when both input types give NaN for NaN model values and the existing metric tests still pass.

Written by the indexing model from the issue text.

Description

nse and r2 return a perfect 1.0 when model values are missing. rmse, bias, mae, kge and cc return NaN for the same input.

import numpy as np, pandas as pd, modelskill as ms

t = pd.date_range("2026-01-01", periods=4, freq="h")
df = pd.DataFrame({"obs": [1.0, 2, 3, 4], "mod": [1.0, 2, np.nan, np.nan]}, index=t)
cmp = ms.from_matched(df, obs_item="obs")
print(cmp.skill(metrics=["nse", "rmse"]).to_dataframe())
#              n  nse  rmse
# observation
# obs          4  1.0   NaN

df["mod"] = np.nan
print(ms.from_matched(df, obs_item="obs").score(metric="nse"))
# {'mod': 1.0}

The same happens calling the metric directly on pandas Series. With NumPy arrays it returns NaN:

obs = pd.Series([1.0, 2, 3, 4])
mod = pd.Series([1.0, 2, np.nan, np.nan])
ms.metrics.nse(obs, mod)                    # 1.0
ms.metrics.nse(obs.values, mod.values)      # nan

Cause

nash_sutcliffe_efficiency uses np.sum on its inputs, which for a pandas Series skips NaN. The numerator then sums only the points where the model has a value, while the denominator sums every observation. The result is not the NSE of the non-missing points either; it overstates the fit, and an all-NaN model scores 1.0. r2 inherits it.

Expected

NaN, as the other metrics return. A model run that failed part-way, and so returns NaN after the failure, should not score as a perfect fit.

Converting the inputs with np.asarray before summing would make it consistent with the other metrics.

modelskill 1.4.0, numpy 2.4.6, pandas 2.3.3. The code is unchanged on main.

Dominant language
Python
Stars
58
Forks
10
Avg merge
28m
Merged PRs (30d)
1

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DHI/modelskill

All issues in DHI/modelskill

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.