Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Validate both sets at fixed validation timestamps (+ ADR 0003, glossary)

Closed
  • Is something wrong?
#142 0 comments 0 reactions 0 assignees View on GitHub

A pull request for this has already been merged.

  • #143 by @debrevitatevitae — merged

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
jupyter-notebook, python
Domain
data, testing

Research direction

Start with the window preparation code and existing tests for validation and test evaluation timestamps; the issue does not name their file paths. Check how prepared data reaches the backbone and posterior stages, HPO, val_crps_baseline, and diagnostics. Implement deterministic, evenly spaced validation window selection using the study's K, then add the property tests and ADR 0003 and commit the glossary change; finish by running the listed checks and confirming a few epoch MLflow run has decreasing losses.

Written by the indexing model from the issue text.

Description

ready-for-agent

Parent

#138

What to build

Part A of #138. Both validation sets (backbone and posterior) are reduced to their validation timestamps: min(K, n) evenly spaced windows per history of n stride-1 windows, including the first and last window. The selection follows the same linspace rule as the test set's evaluation timestamps. K is the value chosen by the K study and is a single protocol constant shared by both sets. Training windows stay stride-1 and untouched. Every consumer (backbone stages, backbone HPO, posterior stages, posterior HPO, val_crps_baseline, the once-only diagnostics) gets the reduced sets through the prepared data, with no change of its own.

Also on this branch:

  • ADR 0003, "Validation at fixed timestamps": the decision, the K-study results, and its consequences:

    • any change of K, or a return to full-set validation, changes every validation score, supersedes ADR 0003, and requires rerunning the backbone and posterior studies;
    • val_crps_baseline uses the validation timestamps;
    • no existing run is invalidated (stages_disjoint_val holds only data generation).

    ADRs 0001 and 0002 stay in force.

  • Glossary: commit the GLOSSARY.md change from grilling (new Validation timestamps entry; Validation criterion scored at the validation timestamps of the posterior validation set). It is currently an uncommitted local change.

Acceptance criteria

  • Property tests (hypothesis) on preparing the windows:
    • every validation history contributes exactly min(K, its window count) windows
    • the windows are evenly spaced, including the first and last
    • they are a subset of the stride-1 windows, in order
    • the selection is deterministic
    • the training windows are identical for any K
    • the validation sets stay disjoint
  • K is logged as a run param; the logged validation window counts reflect the reduced sets
  • The baseline still equals epoch 0's score (ADR 0002 test green)
  • ADR 0003 added; GLOSSARY.md change committed
  • uv run pre-commit run --all-files, pytest, maintained-script smoke tests and a few-epoch mlflow run (losses decreasing) all pass
Dominant language
Jupyter Notebook
Stars
0
Forks
0
Avg merge
1d 2h
Merged PRs (30d)
6

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from GroupiSP/qmodem

All issues in GroupiSP/qmodem

Similar issues

More Data Engineering issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.