bug: PostHog integration export overwhelms target instance with unbounded burst (18k events/sec observed)

Describe the bug

The PostHog integration export can overwhelm the target PostHog instance by dumping all accumulated events in an unbounded burst with no throttling. On a self-hosted Langfuse deployment, we observed 1,857,712 events sent in 103 seconds (~18,000 events/second) during a single hourly sync cycle, triggering PostHog ingestion warnings and likely causing silent event loss.

The root cause is that handlePostHogIntegrationProjectJob runs all export streams (traces, generations, scores, events) in parallel via Promise.all(), each with a separate posthog-node client instance, with no delay between flush cycles.

Evidence from production logs

Normal hourly pattern (~1,200 events, completes in ~6s):

2026-03-25T09:30:00.062Z Sending traces for project to PostHog 2026-03-25T09:30:01.674Z Sent 1192 traces to PostHog 2026-03-25T09:30:02.296Z Sent 1264 generations to PostHog 2026-03-25T09:30:06.568Z Sent 38 scores to PostHog 2026-03-25T09:30:06.574Z PostHog integration processing complete

Catastrophic burst (1.8M events, initial sync or re-sync):

2026-03-24T15:30:00.084Z Executing PostHog Integration Job 2026-03-24T15:30:12.107Z Sent 10000 traces 2026-03-24T15:30:12.442Z Sent 10000 generations ... (3 parallel streams escalating) 2026-03-24T15:31:36.945Z Sent 847152 traces 2026-03-24T15:31:43.543Z Sent 955620 generations 2026-03-24T15:30:19.764Z Sent 54940 scores 2026-03-24T15:31:43.555Z PostHog integration processing complete

PostHog shows $$client_ingestion_warning events with message: "posthog-js client rate limited. Config is set to 10 events per second and 100 events burst limit."

Root cause (code analysis)

In worker/src/features/posthog/handlePostHogIntegrationProjectJob.ts:

1. Parallel execution: All export streams run concurrently via Promise.all(processPromises), creating 2-4 simultaneous posthog-node clients each firing HTTP batches independently.
2. No inter-flush throttle: After each await posthog.flush() (every 10k events), the next batch starts immediately with no delay.
3. flushAt: 1000 sends batches of 1,000 events per HTTP request with no backpressure.
4. Unbounded query: On first sync or re-sync (lastSyncAt = null, defaults to 2000-01-01), the ClickHouse query returns the entire project history with no record limit.
5. Multiple client instances: Each processPostHog* function creates its own new PostHog(...), multiplying concurrent connections.

The existing BullMQ limiter (max: 1, duration: 10_000) only throttles between projects, not within a single project's export.

Suggested fix

1. Run export streams sequentially instead of Promise.all() — reduces concurrent HTTP load from 3-4x to 1x.
2. Add a configurable delay after each flush cycle (e.g., LANGFUSE_POSTHOG_FLUSH_DELAY_MS env var, default 500ms) — gives the target PostHog instance breathing room.
3. Reuse a single PostHog client instance across all export streams — avoids multiple concurrent connection pools.

These changes are backward-compatible and don't affect the data format or export correctness. They also benefit the Mixpanel integration which uses the same pattern.

Steps to reproduce

1. Deploy Langfuse self-hosted with PostHog integration enabled
2. Accumulate a large number of traces (>100k) — or configure PostHog integration for the first time on a project with existing history
3. Wait for the hourly sync at :30
4. Observe the worker logs: all events are sent in a burst with no throttling
5. Check PostHog Data Management → Ingestion Warnings for $$client_ingestion_warning

Langfuse Cloud or self-hosted?

Self-hosted

If self-hosted, what version are you running?

langfuse/langfuse-worker:3.124.1 (also verified against current main — same architecture)

SDK and integration versions

No response

Additional information

• The same pattern exists in the Mixpanel integration (handleMixpanelIntegrationProjectJob.ts) and would benefit from the same fix.
• We are happy to contribute a PR for this fix.

Are you interested in contributing a fix for this bug?

Yes