[Metadata] Reference solutions with how they were obtained: typed references and API

Offen
  • Stimmt etwas nicht?
#409 2 Kommentare 0 Reaktionen 1 zugewiesene Person Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@ocots arbeitet bereits daran.

Seit 07.11.2026.

Bewertung

Dieses Issue wurde noch nicht bewertet.

Beschreibung

feature

Agents: read the guide for agents in #310 first (order, difficulty, how to claim an issue, working method).

Goal

A reference solution says how it was obtained, so that it can be reproduced and used as a benchmark: modeling, modeler, solver and its options (tolerances…), grid, scheme, exit status, iterations, time, constraint violation, platform, package versions, run and commit. The same for every known local solution (#378). An analytic solution of the continuous problem (#379) is a different kind of reference. The package exposes both through one abstract type, concrete types and an API, rebuilt from generated files.

Decisions (maintainer, 2026-11-07, after the triage of #367)

  • Never written by hand. Every field is written by the tests and by tools/references/update.jl; the more information the better, as long as it is generated.
  • No trajectories for the numerical references: too large. The objective and the configuration are enough to reproduce a solve. The trajectories and the plots belong to the results pipeline (#314).
  • An acceptable solve (Ipopt acceptable, JuMP ALMOST_LOCALLY_SOLVED, MadNLP SOLVED_TO_ACCEPTABLE_LEVEL) may be a known solution: it is a reference, with its exit status recorded. Example: moonlander 0.961844 (#367).
  • Analytic references are separate: a continuous solution, with state and control functions, the objective and a source.

Design

1. What the tests record

TestStatus.result! (test/common/status.jl) records, besides what it has today (test/<report>-results.tsv):

  • modeler (empty for JuMP), solver;
  • options: the effective solver options of the solve (defaults of TestConfigs merged with the robustness options), as an inline TOML table in one TSV cell, so that it is read back without a hand-written parser;
  • violation: the primal constraint violation of the solution (OptimalControl: CTModels.constraints_violation; JuMP: from JuMP.primal_feasibility_report).

Each test run also writes test/reports/<report>-environment.toml, once: Julia version, OS and architecture, commit, and the versions of the direct dependencies of the test environment and of the solver binaries (Ipopt_jll). The full Manifest.toml of a run is #314's (env/<run id>/ on the results branch); a reference keeps the run id to find it.

tools/cron/report.py reads the TSV by column name: added columns change nothing there.

2. What update.jl writes

ext/MetaData/references/<problem>.toml keeps every field it has today (the docs of #408 read them) and adds, for each solution of [[<scheme>.solutions]], a table reference: the solve chosen to represent it, with every field of section 1, its platform, run and commit.

The representative of a solution: a strict convergence before an acceptable one, then the smallest violation, then the smallest objective. The objective of the solution is that of its representative, so that value is the objective of a solve that can be redone.

3. Types and API
abstract type AbstractReference end          # contract: objective(ref) (TYP4)

struct AnalyticReference <: AbstractReference  # continuous problem (#379)
    problem; parameters; objective; state; control; source
end

struct NumericalReference <: AbstractReference # one discretized solve
    problem; objective; grid_size; scheme
    modeling; modeler; solver; options        # options::NamedTuple
    status; iterations; time; violation
    platform; run; commit; versions           # versions::NamedTuple
    solves; configurations                    # the solves that reached it
end

Qualified API (no exports, MOD8):

  • OptimalControlProblems.references(problem; grid_size, scheme): the known solutions, best first (empty when none);
  • OptimalControlProblems.best_reference(problem; grid_size, scheme): the first one, or nothing;
  • OptimalControlProblems.analytic_reference(problem, parameters=nothing): or nothing;
  • OptimalControlProblems.objective(ref).

reference_objective, solution_kind and reference_solution_data stay, written on top of them. Two concrete types because their data differ (TYP7).

Order: two pull requests

A cron dispatched on a branch publishes to results and updates the cron issues (#348, #350, #351 came from such a run), so the new references are computed from runs on main:

  1. Record (section 1): tests and TSV, environment file, test/README.md. Merged, then the four crons are dispatched on main (CronOptimalControl, CronJuMP, CronOptimalControlScalarized, CronRobustness).
  2. Store and expose (sections 2 and 3): update.jl, the loader, the types and the API, the references regenerated from those runs, tools/references/README.md.

Relations

  • #314: the results of the last runs (result.json, plots, env/<run id>/) live on results; the references live on main, generated. A problem page shows both: the latest results with their freshness, the references with how they were obtained.
  • #408 (documentation rewrite) reads the TOML files: only fields are added, none renamed or removed.
  • #306: the reference objective and the local solutions of its notes.

Done when

  • PR 1: each row of *-results.tsv has modeler, solver, options, violation; each run writes *-environment.toml; shown on a local run of one problem per modeling.
  • The crons dispatched on main after PR 1, their run ids recorded here.
  • PR 2: every reference file regenerated from those runs; each solution has its reference table; the files are not edited by hand (the consistency test compares them with a regeneration, or checks the generated header and the schema).
  • AbstractReference, AnalyticReference, NumericalReference and the API, tested (including a failing case shown in the PR); reference_objective, solution_kind, reference_solution_data unchanged for their callers.
  • READMEs updated: test/README.md, tools/references/README.md.
Vorherrschende Sprache
Julia
Sterne
13
Forks
1
Ø Merge
3 Std. 9 Min.
Gemergte PRs (30 T.)
47

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus control-toolbox/OptimalControlProblems.jl

Alle Issues in control-toolbox/OptimalControlProblems.jl

Ähnliche Issues

Weitere Issues zu Julia

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.