Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Large dataclips can still cause runs to be lost

Aperta
#1,504 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

@josephjclark ci sta già lavorando.

Dal 25/8/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Despite a lot of improvements to memory handling and payload processing, I think we're still losing lost runs with large dataclips.

I'm looking at impact water which is regularly building 9.8mb dataclips and lost runs.

I'm not sure sure exactly where the error lies - but I think when processing this 10mb dataclip to send to lightning, either the connection is actually timing out or some process (lightning side or worker side) is exceeded a memory limit and giving in.

The worst thing is that this is dying silently and causing the websocket to disconnect, resulting in a lost run

A few things jump to mind:

  • Claude points out the state objects are double encoded in to JSOn strings. bloating the size A 9mb dataclip in memory might acutally serialize to 10mb or more in the event sent to lightning because of the nested JSON escape characters
  • We are not getting good sentry reports when these events timeout - that needs fixing

This is the underlying cause behind https://github.com/OpenFn/kit/issues/1072 and https://github.com/OpenFn/kit/issues/1179. Both those issues describe symptoms of this one underlying thing. They both describe a case where an event fails to send and then bad things happen. This issue is the reason teh events fail.

Lingua principale
TypeScript
Stelle
21
Fork
23
Merge medio
1g 20h
PR unite (30g)
18

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di OpenFn/kit

Tutte le issue di OpenFn/kit

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.