RFC: Adaptive Exponential Backoff and Reconnection Jitter for DDP/WebSocket Connections

Bắt đầu với app/lib/services/connect.ts và app/lib/services/sdk.ts, sau đó xác định cách việc mở lại native của SDK tương tác với ReconnectionManager được đề xuất. Chỉ triển khai tích hợp backoff và jitter sau khi chọn giữa việc chặn các sự kiện kết nối và ghi đè scheduler của SDK. Được xem là hoàn tất khi các lần thử khôi phục tuân theo đường cong độ trễ được đề xuất và kết nối lại thành công sau thời gian máy chủ ngừng hoạt động được mô phỏng, mà không có các lần thử lại đồng bộ.

RFC: Adaptive Exponential Backoff and Reconnection Jitter for DDP/WebSocket Connections

• Type: Resiliency & Infrastructure RFC
• Target Subsystem: Connection Service (app/lib/services/connect.ts) & SDK Initializer (app/lib/services/sdk.ts)
• Status: Proposed

1. Problem Statement & Root Cause Analysis

The client initializes its connection to the Rocket.Chat server via the Rocket.Chat JS SDK:

private initializeSdk(server: string): typeof Rocketchat {
	return new Rocketchat({ host: server, protocol: 'ddp', useSsl: isSsl(server), reopen: __DEV__ ? 20000 : 5000 });
}

The Architectural Defect: Fixed Interval Reconnection

• Reconnection Polling: In production, the reopen configuration parameter is set to a fixed 5000ms (5 seconds). When the connection is dropped, the app blindly retries establishing a connection every 5 seconds.
• Thundering Herd Problem: If a server experiences a temporary outage, or a network flap occurs, thousands of active mobile clients disconnect simultaneously. When they all attempt to reconnect at identical 5-second intervals, they hammer the server. This synchronized reconnection storm creates a distributed denial-of-service (DDoS) effect, preventing the server from recovering gracefully.
• Battery & Data Exhaustion: Mobile clients attempting aggressive WebSocket handshakes every 5 seconds on a flaky cellular network will rapidly deplete device battery and consume unnecessary user cellular bandwidth.

---

2. Proposed Architectural Solution

Replace the static reconnect interval with an Adaptive Exponential Backoff algorithm containing Randomized Jitter.

The Mathematical Model

The delay between reconnection attempts should follow this formula:

$$T_{\text{retry}} = \min(T_{\text{max}}, T_{\text{base}} \times M^{\text{attempt}}) + \text{random_jitter}$$

• $T_{\text{base}} = 1000,\text{ms}$ (Initial retry delay: 1 second)
• $M = 2.0$ (Backoff multiplier)
• $T_{\text{max}} = 60000,\text{ms}$ (Cap the delay at 60 seconds to guarantee connection within 1 minute of recovery)
• $\text{random_jitter} \in [0, 0.5 \times T_{\text{retry}}]$ (Add up to 50% randomized variation to desynchronize clients)

Expected Reconnection Delay Curve

Attempt	Base Delay ($2^n$)	Jitter Range (0–50%)	Total Delay Range
1	1.0s	0.0s – 0.5s	1.0s – 1.5s
2	2.0s	0.0s – 1.0s	2.0s – 3.0s
3	4.0s	0.0s – 2.0s	4.0s – 6.0s
4	8.0s	0.0s – 4.0s	8.0s – 12.0s
5	16.0s	0.0s – 8.0s	16.0s – 24.0s
6	32.0s	0.0s – 16.0s	32.0s – 48.0s
7 (Max)	60.0s	0.0s – 30.0s	60.0s – 90.0s

---

3. Proposed Blueprint & Implementation Strategy

Since the Rocket.Chat JS SDK under the hood uses a simple internal interval, we can either intercept the connection events or override the reconnection scheduler on the SDK driver directly.

Connection Resilience Controller Blueprint

// Proposed Reconnection Controller for app/lib/services/ReconnectionManager.ts

interface IBackoffConfig {
	baseDelay: number;    // 1000ms
	maxDelay: number;     // 60000ms
	multiplier: number;   // 2.0
	jitter: number;       // 0.5 (50%)
}

export class ReconnectionManager {
	private attempt = 0;
	private config: IBackoffConfig;
	private reconnectTimer: ReturnType<typeof setTimeout> | null = null;
	private onReconnectCallback: () => Promise<void>;

	constructor(onReconnect: () => Promise<void>, config?: Partial<IBackoffConfig>) {
		this.onReconnectCallback = onReconnect;
		this.config = {
			baseDelay: 1000,
			maxDelay: 60000,
			multiplier: 2.0,
			jitter: 0.5,
			...config
		};
	}

	public reset() {
		this.attempt = 0;
		if (this.reconnectTimer) {
			clearTimeout(this.reconnectTimer);
			this.reconnectTimer = null;
		}
	}

	/**
	 * Computes the next retry delay using exponential backoff with full jitter.
	 */
	public getNextDelay(): number {
		const rawDelay = this.config.baseDelay * Math.pow(this.config.multiplier, this.attempt);
		const cappedDelay = Math.min(this.config.maxDelay, rawDelay);
		
		// Add randomized jitter
		const jitterRange = cappedDelay * this.config.jitter;
		const randomJitter = Math.random() * jitterRange;
		
		this.attempt++;
		return cappedDelay + randomJitter;
	}

	/**
	 * Schedules the next reconnection attempt.
	 */
	public scheduleReconnection() {
		if (this.reconnectTimer) return;

		const delay = this.getNextDelay();
		console.log(`[Reconnection] Connection lost. Attempt ${this.attempt}. Retrying in ${(delay / 1000).toFixed(2)} seconds...`);

		this.reconnectTimer = setTimeout(async () => {
			this.reconnectTimer = null;
			try {
				await this.onReconnectCallback();
			} catch (error) {
				// If attempt fails, schedule the next one
				this.scheduleReconnection();
			}
		}, delay);
	}
}

Integration in app/lib/services/connect.ts

Modify the socket closure logic to delegate reconnection scheduling to the ReconnectionManager instead of relying on the SDK's built-in hardcoded loop:

// Integration Blueprint inside app/lib/services/connect.ts

import { ReconnectionManager } from './ReconnectionManager';
import { store } from '../store/auxStore';
import { connectRequest } from '../../actions/connect';

let reconnectionManager: ReconnectionManager | null = null;

const performReconnection = async () => {
	const activeServer = store.getState().server.server;
	if (!activeServer) return;
	
	store.dispatch(connectRequest());
	await sdk.current.connect();
};

// Initialize manager
reconnectionManager = new ReconnectionManager(performReconnection);

// Inside the connect() sequence:
connectedListener = sdk.current.onStreamData('connected', () => {
	reconnectionManager?.reset(); // Successful connect resets the backoff state
	// ... normal login logic ...
});

closeListener = sdk.current.onStreamData('close', () => {
	unsubscribeRooms();
	store.dispatch(disconnectAction());
	
	// Trigger the resilient backoff scheduling
	reconnectionManager?.scheduleReconnection();
});

---

4. Implementation Steps

1. Step 1: Add the ReconnectionManager file to the services directory.
2. Step 2: Modify initializeSdk in app/lib/services/sdk.ts to disable native auto-reopening or override it (setting reopen: false or extremely large values).
3. Step 3: Integrate ReconnectionManager into the close stream handler in app/lib/services/connect.ts.
4. Step 4: Test connection drop recovery:
   • Mock server downtime and observe that client reconnection attempts disperse along the backoff delay curve.
   • Verify that when the server comes back online, all clients reconnect without overloading the server.