Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Incident response runbook and incident communication for testnet

Aperta
  • Qualcosa non va?
#167 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@umeradl ci sta già lavorando.

Dal 25/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

documentation Long run

Part of the testnet readiness umbrella (#166).

Why

Item 7 of the pre-testnet requirements research on discussion No. 132, raised again in the Sep 18, 2026 Testnet-Ready session: list in advance what can go wrong during and after launch, and plan the response, so we protect uptime and reputation.

Nothing in the repo covers what happens during an incident. The "public status page" in Developer/issues/validator-operations.md is a different thing: it exists to show validators the network isn't empty before they join.

Scope

A written, rehearsed runbook plus a way to tell external participants what is going on mid-incident. The Security Alliance web3 incident response template is the reference starting point.

  1. Incident catalogue. DIN-specific failure modes, each with severity, detection signal and first response. Starting list:
    • Smart contract bug or exploit in platform contracts (DinCoordinator, DinToken, DinValidatorStake, DinModelRegistry, treasury/fee/emission contracts) or task contracts
    • A GI stuck in a phase with no way forward. This has already happened: Model_0 GI 2 has sat in DINaggregatorsRegistrationStarted since July (see discussion No. 150)
    • Wrong or unexpected slashing
    • Key compromise: DIN-Representative, proxy admin, model owner or validator
    • Malicious or broken service code in a model manifest running on validator nodes
    • Poisoned model updates getting through aggregation
    • IPFS provider or pinning outage, or RPC outage
    • Faucet or test-token drain
  2. Emergency levers. What we can actually do on-chain today. None of the contracts in foundry/src/ has a pause function, so the only lever is a proxy upgrade via UpgradePlatform.s.sol. Decide whether testnet needs a pause/guardian mechanism, or record that we accept not having one.
  3. Roles and contact. Who is on call, who can sign which transactions, and how the team reaches each other fast. The on-call/backup rota itself lives in the internal incident log (item 7).
  4. Participant communication. Where validators, clients and model owners hear about an incident (e.g. the validator Signal group, a status page) and message templates for "investigating / mitigated / resolved".
  5. Post-incident. A short post-mortem template, and how fixes feed back into the DevNet release cycle.
  6. Rehearsal. At least one tabletop run-through with the team before the first external cohort joins.
  7. Internal incident log and maintainer rota. Use discussion No. 170 as the model: it already does this for red or missing develop CI runs. It names backup maintainers, sets an escalation threshold (about 6 hours unaddressed, with more commits on top), opens one thread per incident with the commit hash, run URL and cause, and closes each thread with the fix and a green run, so the discussion doubles as the log. Testnet incidents need the same thing, with one difference: it must be internal, not public.
    • Why not public GitHub: an exploit, key compromise, faucet drain or wrong slashing must not be disclosed before it is mitigated. Discussion No. 170 is fine in public because a red CI run on develop exposes nothing an attacker can use.
    • What it needs: one record per incident (severity, what was detected and when, affected contracts/models/addresses or tx hashes, who is handling it, timeline, resolution, link to the post-mortem). It also needs a named primary and backup maintainer with an escalation rule: who steps in, after how long.
    • Platform (to decide):
      • (a) a private repo in the org, e.g. InfiniteZeroFoundation/incidents, with one issue per incident. Same tooling as today, with access control, labels for severity, and a searchable history.
      • (b) a private Slack channel for real-time coordination, paired with (a) or a doc as the durable log.
      • (c) a private shared doc or tracker (Notion, Google Doc) with a fixed per-incident template.
      • (d) Monitor-driven incident response. OpenZeppelin Monitor (self-hosted, open source) watches DIN contracts for catalogue signals (stuck GI phase, unexpected slashing, faucet drain, proxy admin changes, upgrades) and raises incidents in PagerDuty (free tier, up to 5 users). PagerDuty holds the on-call schedule (primary and backup), the escalation policy and the per-incident record and timeline. Post-mortems are linked from each incident. Contract vulnerabilities are fixed through GitHub private security advisories. SEAL 911 is listed as the external escalation for live exploits. Revisit Hypernative if DIN moves to mainnet with real value at risk.
      • For contract vulnerabilities specifically, also consider GitHub's private security advisories on this repo, which keep the fix and disclosure private until it ships.
    • Public side: once an incident is mitigated, only a sanitized summary goes out through the participant channel (item 4). The internal record stays internal.

Acceptance criteria

  • Runbook doc merged in the repo (location to be decided: Documentation/public/ for the participant-facing part, Documentation/technical/ for the internal procedures)
  • Incident catalogue covers at least the list above, each with a first response
  • Pause/guardian decision recorded
  • Participant communication channel set up and linked from onboarding docs
  • Internal incident log platform chosen and set up (access limited to the team), with primary and backup maintainers named and the escalation rule written down
  • One tabletop rehearsal done, with findings recorded

Timing

Long-running. It must be done before the first external cohort joins testnet, and should be revised after each DevNet release.

Lingua principale
Python
Stelle
4
Fork
7
Merge medio
3g 58m
PR unite (30g)
39

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di InfiniteZeroFoundation/DevNet

Tutte le issue di InfiniteZeroFoundation/DevNet

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.