Proposal: Structured triage and provenance-aware component analysis for ADR detection

Abierto
  • ¿Algo no está bien?
#50 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
32/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
python
Área
ai-infra-agents, security

Línea de trabajo

Comienza revisando el handoff existente de Tier 1/Tier 2 de ADR, el contrato del texto estándar y el flujo de decisión de compatibilidad; el issue no menciona archivos ni pruebas. Compara esos puntos de entrada con el prototipo funcional y la evidencia de desarrollo de ADR-Bench. La tarea estará terminada cuando los maintainers estén de acuerdo sobre el diseño y los límites de contribución antes de enviar los PRs de implementación.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

I would like to propose an extension to ADR Detection that gives Tier 1
a structured way to request deeper component investigation and gives Tier 2
better provenance and decision semantics.

The motivation is to improve recall for security-relevant component behavior
that is not evident from the transcript alone.

I have a working prototype and would like maintainers' feedback on the design
and preferred contribution boundaries before submitting implementation PRs.

Motivation

ADR's fast/slow architecture is a good fit for this problem, but I encountered
three related limitations:

  1. Tier 1 currently provides a mostly binary handoff. It cannot explicitly say
    that component behavior, object lineage, or source provenance is needed to
    resolve the risk.
  2. Source inspection may cover only an entrypoint or invoked component, while
    relevant behavior can reside in imported implementation files or dormant
    registered components.
  3. A single Tier-2 threat bit can conflate two different questions:
    • Is there evidence of a compromised component or malicious scenario?
    • Did the recorded trajectory produce an unauthorized effect?

These distinctions matter for cases where malicious component behavior is
present but was not exercised, an injected instruction was refused, or an
authorized/fixture workflow contains sensitive-looking operations without an
unauthorized effect.

Proposed direction

Structured Tier-1 handoff

Add an optional schema-constrained triage contract that reports:

  • BENIGN_EXIT or ESCALATE;
  • scenario/component risk;
  • injection presence;
  • unsafe plan, action, and observed effect;
  • authorization status;
  • evidence identifiers;
  • whether deeper provenance is required, and why.

Malformed or internally inconsistent output would escalate rather than silently
exit. The existing stock text contract would remain the default.

Bounded, label-free provenance

Allow Tier 2 to inspect source-bound component facts when requested:

  • registered component identity and declared capabilities;
  • source artifact digest;
  • local imported implementation files;
  • structural function/import information;
  • explicit unavailable or incomplete coverage states.

The provider would return no malicious/benign verdict or benchmark label.
Paths and traversal would be constrained, and source content would be treated
as untrusted evidence rather than instructions.

Separate component risk from observed effects

Optionally let Tier 2 report:

  • component_or_scenario_risk
  • observed_unauthorized_effect

Both would use ABSENT, PRESENT, or UNRESOLVED. Existing consumers could
continue receiving one compatibility decision derived from a configured axis.

The component-risk judgment would require concrete source, instruction,
data-flow, privilege, or resource-abuse evidence. Generic randomness, mocks,
empty results, missing source, or ordinary fixture behavior would not be
positive malicious evidence by themselves.

Preliminary development evidence

In a controlled ADR-Bench development comparison using the same Luna model for
both tiers:

Configuration TP FN FP TN Recall FPR Accuracy
Stock ADR/Luna 33 9 30 231 78.6% 11.5% 87.1%
This proposal 40 2 28 233 95.2% 10.7% 90.1%

In our own provenance-flow (not public yet) comparison using the same Luna for both tiers:

Method TP FN FP TN Recall FPR Accuracy
Stock ADR/Luna 59 37 1 95 61.5% 1.0% 80.2%
This proposal 96 0 5 91 100.0% 5.2% 97.4%

Does this problem and direction fit ADR's intended open-source detection architecture?

Lenguaje dominante
Python
Estrellas
2.1k
Forks
230
Merge medio
16 h 8 min
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de uber/ADR

Todos los issues de uber/ADR

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.