apache/fluss

[Feature] Flink Connector support pendingRecords metric

Open

#138 opened on Dec 6, 2024

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Java (570 forks)auto 404
component=connector/flinkgood first issue

Repository metrics

Stars
 (1,979 stars)
PR merge metrics
 (PR metrics pending)

Description

Originally posted by luoyuxia November 28, 2024

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Although we has coverd most of standard metrics defined in FLIP-33: Standardize Connector Metrics, we still miss the metrics of pendingRecords which help us to trace data backlog.

Solution

pending records = latest log offset - fetched log offset.

For snapshot read phase of primary key table, it should be total records in kv snapshot - records emited, but to simplify, we can just trace the phase of incremental reading which read from change log.

Anything else?

No response

Willingness to contribute

  • I'm willing to submit a PR!

Contributor guide

Research direction
Understand the current metrics implementation in Flink connectors for Fluss. Study how pendingRecords is computed as latest log offset minus fetched log offset. For primary key table snapshot phase, consider simplifying to only track incremental reading from changelog. Check existing Flink connector code to see where to add the metric.
Tech stack
java
Domain
backendstream processing
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
JavaFlink connector development
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
50