BMDB nightly: four models give different results run to run, and BIOMD0000000613's is not numerical

Open
  • Is something wrong?
#2,077 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
docker, java
Domain
ci-cd, testing

Research direction

Start from SBMLImporter.hoistInitialConditionToGlobalParameter and Equation.checkInitialCondition, the two sites where BIOMD0000000613's initial condition diverges. Run math generation for 613 and 547 repeatedly in one JVM and across separate JVMs, diffing the resulting MathDescription structure, then repeat inside the nightly-rebuilt Docker image since the report rules out per-run hash ordering. Done means the two failure modes are reproduced locally and one cause (image content, config, or ordering) is pinned down; the separate 1065/547 baseline SKIP-vs-accept decision lives in the nightly workflow config.

Written by the indexing model from the issue text.

Description

The BMDB nightly gate has failed every night since 2026-08-31 (last green: run
33298492096). Each failure is real in the sense that a result differed from the baseline —
but the same models keep changing their minds, so accepting a baseline clears the signal
for a night and not longer.

What the last seven nightlies actually did

model behaviour
BIOMD0000001065 deviated (PASSED) on six of seven nights; matched its FAIL baseline once, on 09-06
BIOMD0000000547 MATH_GENERATION_FAILURE on 09-04/05/06/07; SOLVER_FAILURE on 08-31, 09-01, 09-03
BIOMD0000000613 MATH_GENERATION_FAILURE on 09-03/05/06/07; DIVIDE_BY_ZERO on 09-04
BIOMD0000000457 flipped PASSED → SOLVER_FAILURE on 09-05 only, and came back

Note the 08-30 pass and the 08-31 failure are on the same commit (f0bb594443), which
is what rules out a code change as the cause.

Two of them are numerically borderline, and that is expected

The recorded failures say so:

  • 1065 — CV_REPTD_RHSFUNC_ERR ... pow(u,v) and u=-0.000000<0 and v=2.619640 not an integer, with EN_0_0 = -0.000000. A quantity sitting on zero, and pow of it is only
    defined if the last bit lands positive.
  • 547 — At t = 8.09814e-18, mxstep steps taken before reaching tout. The solver
    stalling at a denormal-scale step.

Nothing about VCell is going to make those two deterministic; they are models balanced on a
knife edge, and the right treatment is probably to stop gating on them rather than to fix
them.

BIOMD0000000613 is different, and is the reason for this issue

It is not arithmetic. It alternates between two failures two phases apart, both about
the same initial condition:

raised where message
some nights writing the solver input divide by zero '(M_initConc / OC_initConc)'
other nights math generation Initial condition for variable 'M' references variable 'L'. Initial conditions cannot reference variables.

In one run the initial condition has been hoisted to the global parameter M_initConc
(SBMLImporter.hoistInitialConditionToGlobalParameter), which then divides by zero. In the
other it still holds a reference to species L, which Equation.checkInitialCondition
rejects outright.

That is the same model, the same input, and the same code producing two structurally
different MathDescriptions.

Correction (see the investigation comment below). I first read this as
ordering-dependent math generation. Running the model ten times in ten separate JVMs
gives the same answer every time, and identity hash codes differ on every JVM start —
so per-run hash ordering is not the mechanism. The variation is between environments, not
between runs, and the leading candidate is now the nightly's nightly-rebuilt Docker
image.

BIOMD0000000547 moving from SOLVER_FAILURE to MATH_GENERATION_FAILURE fits the same
shape: on some runs it reaches the solver, on others math generation refuses it first, with
the same checkInitialCondition message.

Suggested next steps

  1. Find the ordering dependence. Running 613 and 547 through math generation repeatedly
    in one JVM, and across JVMs, should reproduce it without needing the nightly.
  2. Decide what to do about the knife-edge cases. The baseline can say PASS, FAIL or
    SKIP, and none of those means "either outcome is fine". Options: mark 1065 and 547
    SKIP with the reason recorded, the way the workflow's 11 slow models already are; or
    give the baseline a way to accept a set of outcomes.

Today's three deviations are accepted in #2078 so the signal is clear again, with the
reasoning recorded in the commit.

Dominant language
Java
Stars
140
Forks
35
Avg merge
11h 28m
Merged PRs (30d)
19

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from virtualcell/vcell

All issues in virtualcell/vcell

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.