Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Make Writes Idempotent

Aberta
  • Algo errado?
#234 0 comentários 0 reações 0 responsáveis Ver no GitHub

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
32/100
Tipo de issue
Funcionalidade
Clareza
Precisa de esclarecimento
Status de atividade
Ativa
Stack de tecnologia
kafka, python
Domínio
backend, databases, distributed-systems

Direção de pesquisa

Start by tracing the EventGate Lambda path that performs the DB, EventBridge, and Kafka writes, then inspect the existing DB tables and processing-start data. Reproduce or reason through message redelivery after one write fails. Done means redelivery is handled idempotently, arrival timing is documented or captured as needed, and tests or manual evidence cover the scenario.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement
Feature Description

When the same event arrives multiple times, there is no guarantee that it will not be re-inserted into the DB, or re-delivered to our consumers multiple times. There is no idempotency guarantee.

Problem / Opportunity

The EventGate Lambda handling events and performing writes, which is the central to the EventGate system, currently perform triple write: DB, EventBridge, and Kafka. If there is a failure, e.g. Kafka write fails, the Lambda finishes (the two writes are done) but it constructs response payload with HTTP status 500.

Then, the producer is notified about 500 and maybe it stores the msg into DLQ or redelivers the message again immediatelly, doesn't matter - the message is processed again and the triple write happens again - even if it previously succeeded for some of the writes. Therefore, we'll have duplicit items in our DB and our EventBridge/Kafka consumers might receive the message multiple times potentially.

Something like this happened with Unify recently (Unify as consumer), although no producer complained about this. But this certainly is something we should deal with and redesign.

Note: DB tables contain almost no constraints and very little auxiliary timestamp-based columns - like when a message arrives, a new column arrivedAt with default NOW() should be present - an idea - because otherwise we DO NOT KNOW when the message actually arrived (well, in this case, when it was stored in DB, the arrival happens seconds earlier). The processing start, column that is present, is set by producer, not by eventgate system. On producer's retries, this is the same value!

Acceptance Criteria
  • Scenario with msg-redelivery is handled in better and more expected way, idempotency based ideally.
  • Documentation
  • Either programmatic tests exist, or the final solution was tested at least manually and evidence is attached somewhere, maybe into this ticket even.
Linguagem predominante
Python
Estrelas
4
Forks
0
Merge médio
22h 3min
PRs com merge (30d)
8

Preparar o ambiente

  • Inclui um Dockerfile ou arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de AbsaOSS/EventGate

Todas as issues de AbsaOSS/EventGate

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.