Discover and Verify Concepts in Preference and Instruction-tuning Datasets

I have been developing a framework for analyzing Post-Training datas for a while, inspired by the recent Goodfire paper. Happy to hear your thoughts about it!

Goal
Develop a reusable framework for discovering human-readable concepts in the prompts and responses of preference, instruction-tuning, and other post-training datasets. The goal is to understand what a dataset contains and how models behave on it without first defining a fixed list of criteria.

The minimum input is a prompt, a response, and optional metadata:

$$
\mathcal{D}{\mathrm{single}} = {(x_i,y_i,m_i)}{i=1}^{N}.
$$

Here, $m_i$ may contain the generating model, language, data source, or other attributes. A second response and a preference label are optional extensions, not requirements for concept discovery.

Description

The framework learns sparse representations of prompts and responses, assigns candidate natural-language descriptions to the learned features, and verifies those descriptions on held-out examples. It should produce:

• recurring prompt concepts;
• recurring response concepts or behaviors;
• representative examples and counterexamples;
• relationships between prompt concepts and the response behaviors they elicit;
• context-aware model profiles showing which behaviors a model expresses relative to other models.

Sparse autoencoders (SAEs) are a useful starting point because they decompose dense representations into a small set of active features:

$$
z_i =SAE(h_i), \qquad |z_i|_0 \ll M.
$$

BatchTopK SAEs provide one practical architecture, while What's In My Human Feedback? demonstrates that sparse interpretable features can characterize preference datasets and the attributes expressed by annotators.

Goodfire's Anatomy of Post-Training provides a particularly relevant example: its feature- and prompt-conditioned pipelines automatically surface concepts and training examples in Dolci that teach unsafe compliance and other undesirable behaviors. The authors then show that DPO on this data degrades jailbreak robustness on HarmBench and XSTest. The accompanying predictive data-debugging report also traces safety regressions back to individual preference examples. This illustrates how concept discovery can identify actionable problems in post-training data rather than only summarize its topics.

The framework should keep sparse-code computation separate from the downstream analysis:

$$
\mathcal{D} \xrightarrow{\text{sparse-code source}} Z
\xrightarrow{\text{name, verify, relate, report}} \text{concept artifacts}.
$$

The sparse codes may come from an SAE trained over text embeddings or from model-internal activations. Model-specific SAE toolkits are increasingly available: Qwen-Scope, for example, releases SAEs for Qwen3 and Qwen3.5 and applies their features to model analysis, data-centric workflows, and post-training optimization. Supporting such a toolkit should therefore require changing the sparse-code computation and example-level aggregation adapter, while reusing the naming, verification, prompt–response analysis, and reporting stages.

LLM-generated concept names remain hypotheses. A concept should be retained only when an independent verifier recovers its relationship with feature activation on held-out examples; otherwise, the framework should abstain or leave the feature unnamed.

If paired data is available,

$$
\mathcal{D}{\mathrm{pair}} = {(x_i,y{i,1},y_{i,2},\ell_i,m_i)}_{i=1}^{N},
$$

the same concepts can additionally describe how the responses differ and which properties are associated with a particular judge's preference. Such associations should not be presented as objective definitions of good or bad behavior.

Deliverable scope

• A documented input interface for $(x,y)$ records and optional paired preference records.
• A replaceable sparse-code source supporting embedding-based SAEs initially and model-internal SAE codes through adapters.
• Reusable prompt and response concept artifacts containing names, activations, held-out verification results, and example rows.
• Prompt–response concept relationships and context-aware per-model behavior reports.
• At least one comparison with a direct embedding or clustering baseline.
• A reproducible example analysis and public viewer containing a concept map and model reports. The viewer link can be added when its outputs and presentation are ready for external inspection.
• Documentation of failed or unnamed features, prompt dependence, and the observational limits of the analysis.
• Webtool for visualization

Dependencies

• A representative instruction-tuning or preference dataset with stable example identifiers and model metadata where available.
• An embedding model, model activations or existing SAE, and the compute required to obtain sparse codes.
• An LLM endpoint or manual alternative for concept naming and verification.
• Agreement on the held-out verification and feature-selection protocol.
• A stable export schema for connecting the analysis to the public viewer.
• Preference labels are needed only for the optional judge-preference analysis.