Silent data mutation, missing provenance in the compile pipeline and DEA URL (404)

Start with scripts/retrieve_data_from_dea.py and run python scripts/retrieve_data_from_dea.py to reproduce the 404; then inspect compile_cost_assumptions.py and the named pipeline functions. Use snakemake -call --forceall to observe the broader issues. Done requires narrowing these findings into focused changes with validation, provenance, and manual-input behavior covered by tests or an auditable run report.

Checklist

• I am using the current master branch or the latest release v0.15.0

Describe the Bug

Findings from a first-pass review of technology-data v0.15.0, as discussed with @euronion. Listing them together for triage; happy to split into separate focused issues.

1. No validation of output parameters
docs/parameter.csv lists 11 core parameters; the output contains 59. Nothing validates output parameters against a known list, so new or misspelled parameters pass through silently.

2. Hardcoded metadata in compile_cost_assumptions.py
The script holds 6+ hardcoded dicts (dea_sheet_names, uncrtnty_lookup, cost_year_2020, cost_year_2019, source_dict, per-tech usecols). Proposal: move them to a config file, compare against the data at runtime, and report new/missing entries at the end of the run. Same pattern applies to parameters, technologies and units.

3. Imputation is not recorded
.fillna(0), .ffill(), .interpolate() and .mean() are applied silently — the output does not distinguish sourced values from imputed ones. Proposal: tag imputed values in further description or in a dedicated data_quality column.

4. retrieve_data_from_dea.py is broken
It points at the old DEA URL (404). DEA has restructured into 8 category pages under technology-catalogues. The script was never updated (the new URL appears in the v0.14.0 release notes, but data was only downloaded manually) and is not wired into Snakemake.

5. manual_input.csv entries are silently overwritten
add_manual_input uses combine_first so manual values beat DEA data, but later .loc assignments in add_co2_intensity(), add_solar_from_other() and the hardcoded biomass fuel values overwrite manual_input entries unconditionally. These legacy functions predate the manual_input mechanism. Proposal: end the pipeline after data collection + manual input.

6. No per-run data provenance report
Items 1–5 are the same underlying problem: the pipeline mutates data silently. A per-run report covering source coverage, imputations, overrides, unknown parameters and data freshness would make all of it auditable. Combined with dataset versioning, this would let a user reconstruct where any value came from.

Minimal example

Item 4 reproduces on its own:

python scripts/retrieve_data_from_dea.py

The remaining items are observable in the output of a full pipeline run:

snakemake -call --forceall

Error Message

retrieve_data_from_dea.py — the configured DEA URL returns HTTP 404:

None