BIOMD0000001065 flaps between PASS and SOLVER_FAILURE: investigate the fractional-pow-through-zero, and add an INTERMITTENT test status

オープン
#2,005 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
64/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
静か
技術スタック
cpp, java
領域
ci-cd, testing

調査の方向性

Start in vcell-core/.../sedml/testsupport/OmexTestCase.java where Status is defined, then read the FAIL branch in OmexTestReport that compares known_failure_type against execSummary.failure_type; add INTERMITTENT as a fourth value with a parallel branch that accepts either a pass or a matching failure type. For the mechanism, locate the fractional-pow domain guard in SundialsSolverStandalone and print the base with %.17g or check signbit to separate -0.0 from genuine negative drift. Done when BIOMD0000001065 passing or failing with SOLVER_FAILURE is green, while a different failure type still turns the nightly red.

索引モデルが issue の本文から書いたものです。

説明

Testing validation

Summary

BIOMD0000001065 (von Dassow 2000 segment polarity, 148 species) flaps between PASSED and
FAILED (SOLVER_FAILURE). It failed on essentially every nightly from 2026-06-09 to 2026-08-17,
passed once in run 32094317081
(2026-08-18 03:07 UTC), and has failed on every run since.

That single pass was baselined by 52cf62cbc3, which is why the nightly has been red ever since —
full forensics in #2000. This issue is about the model and the mechanism, not that incident.

Is there already a way to mark a flapping test? Half of it exists.

OmexTestCase.Status is a closed enum with exactly three values
(vcell-core/.../sedml/testsupport/OmexTestCase.java:5):

public enum Status { PASS, FAIL, SKIP }

The failure-signature check already exists and works. In OmexTestReport the comparison is not
just pass/fail — for a case documented FAIL, a run that also fails is a change only if the
failure type differs
:

} else if (matchingTestCase.known_status == OmexTestCase.Status.FAIL) {
    if (matchingTestCase.known_failure_type == null
            || !matchingTestCase.known_failure_type.equals(execSummary.failure_type)) {
        // ... recorded as a change

So known_failure_type is exactly the "confirm it failed the way we expect" mechanism. Note also
that known_failure_desc is stored but not compared — correct, since it embeds a per-run temp
path (/tmp/VCell_CLI_<random>/…) and would otherwise differ every night.

What does not exist is a status that tolerates either outcome. Today both directions are
regressions:

baseline run passes run fails
PASS green red
FAIL + type red green if the failure type matches
SKIP red red (recorded, stays SKIP)

So for a genuinely intermittent model there is no setting that is green both ways.

Why the obvious workarounds are not good enough
  • Restore FAIL / SOLVER_FAILURE (the state it held for two months) — correct today and the
    right immediate action, but the gate goes red again the next time it passes: "Test case marked
    as FAIL … but passed"
    . A stopgap that is guaranteed to flap.
  • Mark it SKIP — silences it, but SKIP means not run at all (the model list is also
    hard-coded in NightlyBMDB_CLI.yml, the case $number in 235|255|… block). Total coverage loss:
    we would never learn that it started passing reliably, and never notice if it began failing in a
    new way, which is precisely the signal worth keeping.

Proposed: a fourth status

public enum Status { PASS, FAIL, SKIP, INTERMITTENT }

with the semantics "either outcome is acceptable, but a failure must look like the documented
one"
:

run result INTERMITTENT verdict
passes accepted — no change
fails with known_failure_type accepted — no change
fails with a different failure type change → red

That keeps the model executing and keeps the failure-signature check, while not gating on the coin
flip. It is a small change — one enum value plus a branch in OmexTestReport, alongside the
existing FAIL branch that already does the type comparison.

The mechanism generalises beyond this model; splitting it into its own issue is reasonable if it
grows.

The actual question about this model

Marking it intermittent manages the symptom. Why it is intermittent is the interesting part, and
#2000 has narrowed it considerably:

  • The generated MathDescription is byte-identical across the shas either side of the flip
    (verified, with a negative control) — no VCell model-processing change is involved.
  • The solver binary is unchanged (v0.0.44-dev4, asset last uploaded 2024-02-11).
  • The corpus is unchanged (and now pinned by #2003).

The failure is a Hill term raising a species to a fractional power as it passes through zero:

pow(u,v) and u=-0.000000<0 and v=2.619640 not an integer
  where: EN_0_0 = -0.000000

Two candidate explanations, with different fixes:

  1. The guard is wrong. If EN_0_0 is true IEEE -0.0, then -0.0 < 0 is false and C99
    §F.9.4.4 gives pow(-0.0, 2.61964) = +0.0. The C++ guard in SundialsSolverStandalone would be
    rejecting a legal input — and this becomes an instance of #1646, currently marked Done on the
    board. If so, fixing it may make this model pass reliably, which would be the real fix and
    would explain the intermittency as "sometimes the drift lands exactly on -0.0, sometimes a
    hair below".
  2. It is drift, and the model is simply marginal. A stiff 148-species model whose integrator
    undershoots zero into a fractional power will hit this eventually; whether it does depends on
    last-ulp differences, which vary with the runner's CPU (glibc pow dispatches differently with
    and without FMA). Then the question is whether VCell should clamp a negative concentration to
    zero before a fractional pow — a modelling decision, not just a guard fix.

Distinguishing them is cheap: print the base with %.17g, or test signbit, at the guard.

Suggested work

  1. Settle (1) vs (2) — one print statement in the solver's domain guard.
  2. Restore 1065 to FAIL / SOLVER_FAILURE so the nightly is honest in the meantime (#2000).
  3. Add INTERMITTENT, and apply it here.
  4. If it turns out to be (1), fix the guard and check whether this model — and any others using
    fractional powers of concentrations — become reliable.

Worth grepping the corpus for other models with fractional exponents on species; if the guard is
wrong, 1065 is unlikely to be the only one sitting near the edge.

Refs #2000, #1646, #1901

主要言語
Java
スター
140
フォーク
35
平均マージ
22時間 6分
マージ済み PR(30日)
14

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

virtualcell/vcell のほかの issue

virtualcell/vcell の issue をすべて見る

似ている issue

Java の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。