Missing simulation datasets: 5 confirmed losses on published models, all 2004-2005 — find the extent and the cause

Abierto
  • ¿Algo no está bien?
#1,980 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
50/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
java, shell, sql
Área
databases

Línea de trabajo

Start from the sweep described in the issue: query vc_simulationjob for hasData='Y' and check files under /simdata, /simdata_secondary, /share/apps/vcell12/users from the prod data pod, stratifying results by year to confirm or break the 2004-2005 concentration. Then retest the PARENTSIMREF signal against simulations matched on publication status and year, and check DbDriver.java:1711 and any vcell5-era migration or cleanup tooling for what could have removed 2004-2005 shares. Done means a dated extent of the loss, a supported cause hypothesis, and a recommendation for the five rows (re-run or mark unavailable).

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Focused follow-up to #1966, which established the infrastructure facts (cfs07 now mounted, 904 of 907 "missing" simulations never had results) and is closed. This issue is only about the datasets that are genuinely gone: how many, and why.

What is known

A simulation with a vc_simulationjob row carrying hasData='Y' is the database asserting that results exist. Where the files are absent from every share, that assertion is wrong and data has been lost.

Sweeping every such simulation belonging to a published BioModel — 1747 across 258 published models, checked against /simdata (cfs09), /simdata_secondary (cfs07) and /share/apps/vcell12/users (cfs12) from inside the prod data pod:
data present 1742 (99.7%)
missing 5
user simId year bioModel parentSimRef
mayya 10480740 2004 10829774 (STATmodulePaper) set
mayya 10483149 2004 10829774 set
mayya 10483210 2004 10829774 10415851
mayya 10633806 2004 10829774 10633707
partharoy 12119710 2005 12119723 (7_12_00_model1) 12117313

Zero files of any kind survive for these — not just the .log the data service looks for first.

Three patterns, none yet explained

  1. Time. All five are 2004–2005. Published-model simulations with hasData='Y' span 2002–2026, and 84 of them predate 2006 — so the losses are ~6% of the pre-2006 population and 0% of everything after. Whatever happened appears to have stopped.

  2. Concentration. Four of five are on one BioModel (10829774), owned by one user. That model has 32 simulations, all claiming data; 28 are intact. So this is partial loss within a model, not a model-level deletion.

  3. Lineage. All five have a non-null PARENTSIMREF, against a base rate of 1229 / 261,111 = 0.47% among simulations with hasData='Y'. Under that base rate, 5-for-5 by chance is vanishingly unlikely.

    But the selection is not clean. These came from published models, which are both more crawled (hence noticed) and plausibly more copied (hence more likely to have parents). The correlation may be with publication, not with having a parent. Test this before building on it.

    The parents themselves — 10415851, 10633707, 12117313 — also carry hasData='Y' and are also missing. So whatever removed the child removed the parent; it is not a copy-versus-original distinction.

What this rules out

  • Not ResultSetCrawler retention logic. #1966 disconfirmed this twice: every affected simulation has a direct vc_biomodelsim link (none is parent-ref-only), and getSimulationInfos(user, false) resolves to WHERE ownerRef = <user> with no join to the link tables (DbDriver.java:1711), so a row is retained regardless of how it is referenced.
  • Not the missing cfs07 mount. That is now fixed and verified, and these files are absent from cfs07 as well.
  • Not wholesale storage loss. User directories are intact — les 63,769 files, JDitlev 50,327, prangamani 41,834 — and 99.7% of published-model results are present.

Proposed work

  1. Extend the sweep beyond published models. The 1747 checked here are the models we care most about, but they are 0.7% of the 261,111 simulations with hasData='Y'. A full sweep is a few hours of ls; a stratified sample by year would answer "is this era-bound?" much faster. The 2004–2005 concentration is the thing to confirm or break first — if the rate outside published models is also ~0% post-2005, that dates the event.
  2. Test the PARENTSIMREF correlation properly. Compare against simulations matched on publication status and year. If it survives that, it is a lead; if it does not, it was publication all along.
  3. Find out what happened in 2004–2005. Storage migrations, a vcell5-era share retirement, or an early cleanup tool are all candidates. vcell5 (2.3P, 1.4P used) is mounted nowhere today and was searched in the original sweep — worth asking whether it ever held this era's data and what moved off it.
  4. Decide what to do about the five. They belong to published papers, so the results are likely reproducible by re-running the simulations, and the models themselves are intact. Options: re-run and re-attach, or mark them honestly as unavailable. Doing nothing means a user opening a published model gets an error with no explanation.
  5. Consider making the assertion self-checking. hasData='Y' with no files is a database claim contradicted by the filesystem, and nothing notices until a user asks. A periodic reconciliation over published models — 1747 rows, cheap — would turn silent loss into a report.

Not urgent, and worth saying why

Five datasets out of 1747 on published models, none newer than 2005, on a system where the corresponding models are all still present. This is a data-integrity question worth answering, not an outage. The user-facing symptom that made it visible (#1967) is already fixed and ships to prod with 8.0.25.01.

Lenguaje dominante
Java
Estrellas
140
Forks
35
Merge medio
22 h 6 min
PR fusionados (30 d)
14

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de virtualcell/vcell

Todos los issues de virtualcell/vcell

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.