Add change detection and watermark tracking to extractors

Inizia leggendo le interfacce dei plugin in plugins/plugin.go e l’orchestrazione del runner in runner/runner.go, quindi esamina runner/config.go, cmd/run.go e l’estrattore BigQuery. Definisci i confini di state/watermark e come mantenere disponibile l’estrazione completa. Il lavoro è completato quando gli estrattori ad alto volume elencati possono emettere metadati delle modifiche e ricorrere all’estrazione completa quando non esiste alcun watermark.

Context

Meteor currently extracts everything on every run. For high-volume sources, this is wasteful and slow. Change detection allows extractors to emit only what changed since the last run.

Scope

• Define a watermark/checkpoint interface that extractors can implement
• Store watermarks between runs (local file, config store, or Compass state)
• Update high-volume extractors to support incremental extraction:
  • BigQuery — use INFORMATION_SCHEMA timestamps for modified tables
  • Postgres/MySQL — track schema modification timestamps
  • Kafka — track topic configuration changes
• Emit change type metadata on records: created, updated, deleted
• Fall back to full extraction when no watermark exists

Design Considerations

• Watermark storage should be pluggable (local file for dev, remote store for production)
• Full extraction should remain available as a fallback or explicit mode
• Change detection accuracy varies by source — document limitations per extractor

Key Files

Why

Incremental extraction reduces load on sources, shrinks payloads, and enables faster refresh cycles. A graph that updates in minutes instead of hours is significantly more useful for AI agents.

References

• Meteor Roadmap — Incremental and Event-Driven