Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Incident response runbook and incident communication for testnet

Abierto
  • ¿Algo no está bien?
#167 1 comentario 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@umeradl ya está trabajando en esto.

Desde el 25/9/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

documentation Long run

Part of the testnet readiness umbrella (#166).

Why

Item 7 of the pre-testnet requirements research on discussion No. 132, raised again in the Sep 18, 2026 Testnet-Ready session: list in advance what can go wrong during and after launch, and plan the response, so we protect uptime and reputation.

Nothing in the repo covers what happens during an incident. The "public status page" in Developer/issues/validator-operations.md is a different thing: it exists to show validators the network isn't empty before they join.

Scope

A written, rehearsed runbook plus a way to tell external participants what is going on mid-incident. The Security Alliance web3 incident response template is the reference starting point.

  1. Incident catalogue. DIN-specific failure modes, each with severity, detection signal and first response. Starting list:
    • Smart contract bug or exploit in platform contracts (DinCoordinator, DinToken, DinValidatorStake, DinModelRegistry, treasury/fee/emission contracts) or task contracts
    • A GI stuck in a phase with no way forward. This has already happened: Model_0 GI 2 has sat in DINaggregatorsRegistrationStarted since July (see discussion No. 150)
    • Wrong or unexpected slashing
    • Key compromise: DIN-Representative, proxy admin, model owner or validator
    • Malicious or broken service code in a model manifest running on validator nodes
    • Poisoned model updates getting through aggregation
    • IPFS provider or pinning outage, or RPC outage
    • Faucet or test-token drain
  2. Emergency levers. What we can actually do on-chain today. None of the contracts in foundry/src/ has a pause function, so the only lever is a proxy upgrade via UpgradePlatform.s.sol. Decide whether testnet needs a pause/guardian mechanism, or record that we accept not having one.
  3. Roles and contact. Who is on call, who can sign which transactions, and how the team reaches each other fast. The on-call/backup rota itself lives in the internal incident log (item 7).
  4. Participant communication. Where validators, clients and model owners hear about an incident (e.g. the validator Signal group, a status page) and message templates for "investigating / mitigated / resolved".
  5. Post-incident. A short post-mortem template, and how fixes feed back into the DevNet release cycle.
  6. Rehearsal. At least one tabletop run-through with the team before the first external cohort joins.
  7. Internal incident log and maintainer rota. Use discussion No. 170 as the model: it already does this for red or missing develop CI runs. It names backup maintainers, sets an escalation threshold (about 6 hours unaddressed, with more commits on top), opens one thread per incident with the commit hash, run URL and cause, and closes each thread with the fix and a green run, so the discussion doubles as the log. Testnet incidents need the same thing, with one difference: it must be internal, not public.
    • Why not public GitHub: an exploit, key compromise, faucet drain or wrong slashing must not be disclosed before it is mitigated. Discussion No. 170 is fine in public because a red CI run on develop exposes nothing an attacker can use.
    • What it needs: one record per incident (severity, what was detected and when, affected contracts/models/addresses or tx hashes, who is handling it, timeline, resolution, link to the post-mortem). It also needs a named primary and backup maintainer with an escalation rule: who steps in, after how long.
    • Platform (to decide):
      • (a) a private repo in the org, e.g. InfiniteZeroFoundation/incidents, with one issue per incident. Same tooling as today, with access control, labels for severity, and a searchable history.
      • (b) a private Slack channel for real-time coordination, paired with (a) or a doc as the durable log.
      • (c) a private shared doc or tracker (Notion, Google Doc) with a fixed per-incident template.
      • (d) Monitor-driven incident response. OpenZeppelin Monitor (self-hosted, open source) watches DIN contracts for catalogue signals (stuck GI phase, unexpected slashing, faucet drain, proxy admin changes, upgrades) and raises incidents in PagerDuty (free tier, up to 5 users). PagerDuty holds the on-call schedule (primary and backup), the escalation policy and the per-incident record and timeline. Post-mortems are linked from each incident. Contract vulnerabilities are fixed through GitHub private security advisories. SEAL 911 is listed as the external escalation for live exploits. Revisit Hypernative if DIN moves to mainnet with real value at risk.
      • For contract vulnerabilities specifically, also consider GitHub's private security advisories on this repo, which keep the fix and disclosure private until it ships.
    • Public side: once an incident is mitigated, only a sanitized summary goes out through the participant channel (item 4). The internal record stays internal.

Acceptance criteria

  • Runbook doc merged in the repo (location to be decided: Documentation/public/ for the participant-facing part, Documentation/technical/ for the internal procedures)
  • Incident catalogue covers at least the list above, each with a first response
  • Pause/guardian decision recorded
  • Participant communication channel set up and linked from onboarding docs
  • Internal incident log platform chosen and set up (access limited to the team), with primary and backup maintainers named and the escalation rule written down
  • One tabletop rehearsal done, with findings recorded

Timing

Long-running. It must be done before the first external cohort joins testnet, and should be revised after each DevNet release.

Lenguaje dominante
Python
Estrellas
4
Forks
7
Merge medio
3 d 58 min
PR fusionados (30 d)
39

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de InfiniteZeroFoundation/DevNet

Todos los issues de InfiniteZeroFoundation/DevNet

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.