Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

Make Writes Idempotent

Offen
  • Stimmt etwas nicht?
#234 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
32/100
Issue-Typ
Feature
Klarheit
Muss geklärt werden
Aktivitätsstatus
Aktiv
Tech-Stack
kafka, python
Bereich
backend, databases, distributed-systems

Rechercherichtung

Start by tracing the EventGate Lambda path that performs the DB, EventBridge, and Kafka writes, then inspect the existing DB tables and processing-start data. Reproduce or reason through message redelivery after one write fails. Done means redelivery is handled idempotently, arrival timing is documented or captured as needed, and tests or manual evidence cover the scenario.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

enhancement
Feature Description

When the same event arrives multiple times, there is no guarantee that it will not be re-inserted into the DB, or re-delivered to our consumers multiple times. There is no idempotency guarantee.

Problem / Opportunity

The EventGate Lambda handling events and performing writes, which is the central to the EventGate system, currently perform triple write: DB, EventBridge, and Kafka. If there is a failure, e.g. Kafka write fails, the Lambda finishes (the two writes are done) but it constructs response payload with HTTP status 500.

Then, the producer is notified about 500 and maybe it stores the msg into DLQ or redelivers the message again immediatelly, doesn't matter - the message is processed again and the triple write happens again - even if it previously succeeded for some of the writes. Therefore, we'll have duplicit items in our DB and our EventBridge/Kafka consumers might receive the message multiple times potentially.

Something like this happened with Unify recently (Unify as consumer), although no producer complained about this. But this certainly is something we should deal with and redesign.

Note: DB tables contain almost no constraints and very little auxiliary timestamp-based columns - like when a message arrives, a new column arrivedAt with default NOW() should be present - an idea - because otherwise we DO NOT KNOW when the message actually arrived (well, in this case, when it was stored in DB, the arrival happens seconds earlier). The processing start, column that is present, is set by producer, not by eventgate system. On producer's retries, this is the same value!

Acceptance Criteria
  • Scenario with msg-redelivery is handled in better and more expected way, idempotency based ideally.
  • Documentation
  • Either programmatic tests exist, or the final solution was tested at least manually and evidence is attached somewhere, maybe into this ticket even.
Vorherrschende Sprache
Python
Sterne
4
Forks
0
Ø Merge
3 T. 7 Std.
Gemergte PRs (30 T.)
11

Entwicklungsumgebung

  • Enthält ein Dockerfile oder eine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus AbsaOSS/EventGate

Alle Issues in AbsaOSS/EventGate

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.