Technique apps: the batch as the main page, and inspecting a value before a long reduction

Start with the parked 24-sans-batch branch and read #159 and #160, which the proposed first step depends on; docs/requirements/diffraction.md gives a comparison case. The issue proposes a technique app page and value inspection flow but leaves several design questions open. Done would require settling those questions and building the page with masks and beam centre as preparation steps.

This issue describes the direction for technique apps, such as the SANS app of #24: how a page is organised, and how a person inspects a value before a long reduction starts. Most of it is a proposal to think through, not a plan. Nothing here has been built.

The input is a comparison with ess-sans-widgets, an ipywidgets and Voilà prototype from the instrument side. The comparison used its code and one screenshot, without running it. It shows how one scientist thinks about the task, so it is evidence, not a requirement.

Decided

• The batch view is the main page of a technique app. The generic tabs (Workflows, Outputs, Records) stay behind it, for inspecting or redoing a step. Visiting users should not need to learn framework concepts for the common task.
• A person inspects a value before a long reduction starts. The value comes from a preparatory run, a file, or SciCat, and the reduction is one run or a batch. This shape is likely for many apps, whether or not the beam centre itself turns out to need it.
• A beam centre can be shared or per sample. A sample changer may reuse one centre, and a more invasive change of sample can need its own (#24).

How the page is organised today

ess-sans-widgets                       essapps generic app
────────────────────────────           ────────────────────────────
Settings (shared, one accordion)       [Workflows] Outputs  Records
 folders, files, instrument,            spec picker, Submit
 bank, λ and Q bins, gravity            one form per spec:
[Manual] Smart  Auto                     one parameter tab at a time
 how the rows get made                  message
 buttons                               Outputs: table + plot/provenance
 table of runs (CSV text)              Records: table
 log
 plots, added as rows finish

ess-sans-widgets is organised around a batch of samples. The generic app is organised around framework objects, which suits a generic app but not the main page of a technique app.

Proposed page

SANS tab
 Prepare
   Masks        [form] [Compute]  completed   view: Masks on image    [Use]
   Beam centre  [form] [Compute]  completed   view: Centre on image   [Use]
 Settings of all runs   masks: Masks (11:45)   beam centre: BeamCentre (11:52)
 Runs                   table: run, transmission, status
 [Reduce all]           results

What this takes from ess-sans-widgets:

1. Shared settings and per-row values are separate on screen. This is Template: fixed params as settings, blanks as table columns. Which parameters are columns should vary per batch, since a per-sample centre makes the centre a column. The parked 24-sans-batch has Form(IOFQ, client, blanks=_ROW), but _ROW is fixed.
2. One table with several sources of rows. Manual, Smart, and Auto in ess-sans-widgets are one table filled from a person's CSV, a folder scan, or a scan on a timer. The parked Batch has only the scan (batch.rows). Auto corresponds to TriggerLoop (#160), which runs outside the page.
3. Results next to their input. The order of use in the LoKI notebook moves between Workflows and Outputs at each step. The parked Batch already draws I(Q) in place.
4. All settings visible at once. The left-hand tabs of Form show one parameter at a time, so checking a request before submitting takes many clicks. At least a read-only summary would help.
5. Folders chosen in the app. Today FolderFiles is passed to local() before App exists. This is the "Working folder" row of #155.

What not to take from it:

• CSV in a text box. Errors cannot point at a cell, and there is no per-row status. SubmitError positions and records can give both.
• A log as the only status, and no history. Each run clears the log and plots, and output files do not record the parameters used.
• Settings read while Auto runs. An edit silently changes later samples. A fixed template avoids this.
• The loop inside the page. It ends with the kernel. In essapps it belongs in a service, and the page shows what it did.

Inspecting a value first

Proposed rule: every value a person must inspect is an output of a record, and picking it for the reduction is what accepts it.

An app declares only its steps (Prepare and SansTab are new names here):

SansTab(
    IOFQ,
    prepare=[Prepare(MASKS, output='masks', into='masks'),
             Prepare(BEAM_CENTRE, output='centre', into='beam_centre')],
    row=('sample', 'sample_transmission'),
)

Use copies record.ref(output) into the settings form, and into any later step's field of that type, so the beam-centre form gets the masks. Reduce all stays refused until those fields are set, since a required field with nothing given is already refused.

One rule covers each source of a value:

Source	How it becomes a record
Preparatory run	It already is one.
File	A small load spec, as MASKS does: it reads XML files and outputs PixelMasks.
SciCat	The same load spec, once the dataset is listed as a file (story C2, not implemented).
Earlier result	listing.fitting(T) already offers it.
Value typed in	There is no run to load. If it should be drawn, a spec takes the value and plots it on an image.

Plotting a file directly in the client is ruled out. A client of a hosted backend cannot read the file, and the plot would not be part of the provenance. Going through a record gives provenance back to the file with no extra code.

Per sample. The same step works per row. Adding beam_centre to row makes the centre a column:

Run     Transmission  Centre        I(Q)
60395   60394         completed ✓   pending
60387   60386         failed        -

"Compute centres" submits one record per row under member, selecting a row shows its view, and Use accepts it. Whether the centre is shared or per sample is then whether it is a setting or a column.

What exists. Spec types say which fields can be prepared, such as Referenced[PixelMasks]. Plots gives the views of a record by spec. Form(blanks=) and App(tabs=) are on the parked 24-sans-batch, not on main.

What is missing.

• Prepare and its layout, in the app layer. The state behind them has no widget dependency, so it belongs in the shared UI core (#150).
• A way to set one field of a form. Form.set resets each field it is not given.
• A choice of view at a step: Plots is keyed by spec, and a step may have several views.
• A load spec for direct_beam, which is a plain file field today, and a Referenced[...] field type for it.

Left out on purpose.

• An "approved" state. The reference in the request is the acceptance, and provenance shows which value was used. It does not record that anyone looked at it.
• Picking the newest value automatically, which would remove the check.

Open questions

1. Can a person skip a step, so that Reduce all computes it? ess-sans-widgets never shows the centre, and the proposal above always asks for it. A reduction a person trusts may not need the extra clicks.
2. Is a trial row a preparatory step? A person may want to reduce one sample, look at it, and then reduce the rest. The parked Batch can edit and submit one row, but nothing ties that to Reduce all.
3. Does anything need to record that a person looked at a value, such as for audit? No requirement says so today.
4. How does the page show automatic reduction? A rule's status lives in the TriggerLoop object, which another process cannot read (#151). A rule can also hold a reference that a person accepted, but that needs a way to pass it to the service.
5. Does a settings panel need to feed several specs of a chain? A Form holds one spec. This depends on how much the settings of the LoKI specs overlap, which has not been checked.
6. Does changing the folder while the app runs need a new backend, or can a file source change? Records name files by identity, so this needs a decision.
7. Does the shape fit another technique? docs/requirements/diffraction.md says vanadium peak removal is something "a human should inspect", and the result is saved and reused for sample runs. This looks like the same shape, with the saved file reloaded through a load spec. It needs checking with someone who knows diffraction.

Possible first step

Build the page on 24-sans-batch, with masks and centre as the two steps, after #159 and #160 are merged. That branch already has the settings form and the table of runs, and #24 needs that app.

Related: #24, #150, #151, #155, #160.