PHM task set: RUL records are graded against C-MAPSS specs, including for bearing datasets with no run-to-failure data

Start by reading data/task/phm_utterance.jsonl and inspect the referenced pdm_rul_*, pdm_cost_benefit_002, and pdm_safety_007 records, comparing each utterance with its characteristic_form. Check which named datasets provide run-to-failure data and whether their grading targets match those sources. Done means the mismatched specs and unsupported RUL tasks are corrected or reclassified, with the affected records validated for consistency.

Summary

In the Hugging Face dataset ibm-research/AssetOpsBench, config prognostics_and_health_management (data/task/phm_utterance.jsonl, 75 records), several RUL Prediction records are graded against a characteristic_form that describes a different dataset than the utterance asks about — in most cases a C-MAPSS turbofan spec attached to a bearing-dataset prompt. A related group asks for remaining-useful-life prediction from bearing datasets whose public releases contain no run-to-failure data and therefore no RUL labels.

Every record referenced below carries "deterministic": true, i.e. it is declared exactly verifiable against ground truth.

Issue 1 — characteristic_form names a different dataset than the utterance

id	utterance asks about	characteristic_form specifies
pdm_rul_008	Paderborn	CMAPSS_FD003
pdm_rul_011	IMS	CMAPSS_FD001
pdm_rul_012	HUST	CMAPSS_FD002
pdm_rul_013	MFPT	CMAPSS_FD003 (MFPT is never named in the spec)
pdm_rul_014	Paderborn	CMAPSS_FD004

Two of these contradict themselves within a single sentence:

• pdm_rul_014: "The process initiates by using load_dataset to acquire the CMAPSS_FD004 training and test splits from the Paderborn dataset"
• pdm_rul_008: "The evaluation workflow for multimodal sensor analysis on the CMAPSS_FD003 dataset … begins with data ingestion using load_dataset to import the Paderborn multimodal training and test sets"

And pdm_rul_012: "The workflow begins by using load_dataset to access the CMAPSS_FD002 HUST train and test splits".

Issue 2 — RUL is requested from datasets with no run-to-failure data

CWRU, Paderborn (KAT) and MFPT are published as fault-classification benchmarks. Their releases are short recordings at fixed speed/load with a known pre-existing defect, labelled by damage type and severity — not continuous degradation trajectories, and with no per-unit time-to-failure label:

• CWRU: single-point faults introduced by electro-discharge machining at defined diameters (0.007″–0.040″), ~10 s records at 12 kHz / 48 kHz, motor load 0–3 hp, near-constant shaft speed.
• Paderborn (KAT): 32 bearings — 6 healthy, 12 artificially damaged, 14 damaged in accelerated lifetime tests — distributed as repeated 4 s measurement segments per operating point. Lessmeier et al. (PHM Europe 2016) present it explicitly as a benchmark for data-driven classification. The lifetime tests themselves are not released as per-unit life trajectories. The same file acknowledges this framing in pdm_fault_006, which asks the agent to distinguish artificial from natural damage.
• MFPT: baseline, outer-race and inner-race fault conditions under varying load, plus three real-world fault cases. No life trajectories.

Records requiring RUL from these sources:

id	dataset	what the spec requires
pdm_rul_002	CWRU	train_rul_model → predict_rul → load_ground_truth → plot_degradation_trends; MAE/RMSE against "actual failure times"
pdm_rul_007	MFPT	train_rul_model → predict_rul → load_ground_truth; MAE/RMSE
pdm_rul_008	Paderborn	same, plus a per-sensor RUL ablation
pdm_rul_013	MFPT	same
pdm_rul_014	Paderborn	same
pdm_cost_benefit_002	CWRU	"RUL prediction generation to quantify operational risk across all test units"
pdm_safety_007	CWRU	predict_rul for all test units; MAE 10–25, RMSE 15–30

The run-to-failure sources in the same file (XJTU-SY, FEMTO/PRONOSTIA, IMS, C-MAPSS) can support RUL tasks; Issue 2 is specific to the classification-only sources.

Evidence that pdm_rul_002 (CWRU) carries a C-MAPSS ground truth

pdm_rul_002 never names C-MAPSS, but its grading targets trace back to one:

1. Target MAE 16.2–32.3 and RMSE 21.6–43.1 appear in exactly two records in the file: pdm_rul_002 (CWRU) and pdm_rul_012 — whose own spec names CMAPSS_FD002.
2. The expected top-5 risk units, "units such as 6, 10, 43, 176, and 210", are identical in pdm_rul_002 (CWRU) and pdm_rul_007 (MFPT), and all five appear in pdm_rul_012's enumerated CMAPSS_FD002 test ids ("6, 10, 43, 176, 210, 115, 123, 137, 89, and 101"). Unit indices up to 210 are consistent with C-MAPSS FD002's 259 test units; CWRU has no comparable set of life-tracked units.
3. The spec trains on features named vibration_horizontal and vibration_vertical. A horizontal/vertical accelerometer pair is the layout of the run-to-failure rigs (XJTU-SY, PRONOSTIA/FEMTO), not of CWRU, whose channels are drive-end, fan-end and base accelerometers (DE, FE, BA) mounted at the 12 o'clock position.

Why this matters

1. With deterministic: true on a target that the named dataset cannot produce, an agent cannot pass by behaving correctly — the reachable ceiling is emitting CWRU "RUL" numbers that happen to land in a C-MAPSS FD002 error band.
2. The correct behaviour — declining to regress RUL from a seeded-fault dataset and explaining why — scores as failure. These records invert the signal for precisely the physical-validity reasoning the benchmark aims to measure.
3. Comparability: scores on the PHM config mix well-posed prognostics tasks with ones whose ground truth belongs to a different dataset.

Suggested resolution

1. Retarget the CWRU / Paderborn / MFPT RUL records to a run-to-failure source (XJTU-SY, FEMTO/PRONOSTIA, IMS, NASA/IMS bearing), or reclassify them as fault classification, severity estimation, or health-index tasks.
2. Rebuild the five mismatched characteristic_form entries from the dataset each utterance actually names, dropping the carried-over FD00x error bands and unit ids.
3. Replace the horizontal/vertical channel names in pdm_rul_002 with the channels of whichever dataset the record ends up targeting.
4. Consider a validation check over phm_utterance.jsonl asserting that (a) a record's characteristic_form names only the dataset its text names, and (b) category: RUL Prediction is paired only with a run-to-failure source.

Questions

• Is the PHM harness published anywhere? The tools these specs reference (load_dataset, load_ground_truth, train_rul_model, predict_rul, verify_ground_truth, calculate_mae, plot_degradation_trends) do not appear in IBM/AssetOpsBench on main or on any of the 149 remote branches as of 2026-10-25. If the config is currently spec-only, that would explain why these records have not surfaced in execution.
• Dataset identification. pdm_rul_009 / pdm_rul_015 refer to "the Mendeley dataset". Mendeley Data is a host, not a dataset; pdm_rul_015's spec resolves it to FEMTO, while pdm_rul_009 describes bearings under 600–1800 RPM speed sweeps, which does not match FEMTO. Similarly, could you confirm which HUST dataset pdm_rul_006 / pdm_rul_012 target? The commonly used HUST bearing set is also a classification dataset, which would place those two records under Issue 2 as well.

---

Happy to open a PR against phm_utterance.jsonl for whichever direction you prefer — retargeting the datasets or recategorising the tasks.