apache/pinot

Error Masked in Peer Server Segment Finder

Open

#14276 opened on Oct 22, 2024

View on GitHub
 (4 comments) (0 reactions) (1 assignee)Java (4,937 stars) (1,234 forks)batch import
good first issue

Description

We often get error segments due to the inability of one of the replicas to download an online segment from a peer.

And most of the times, we aren't able to see a clear error message for this. Stack trace looks something like the following:

org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 5 attempts
     at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65)
     at org.apache.pinot.core.util.PeerServerSegmentFinder.getPeerServerURIs(PeerServerSegmentFinder.java:81)
     at org.a
pache.pinot.core.util.PeerServerSegmentFinder.getPeerServerURIs(PeerServerSegmentFinder.java:67)
     at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.lambda$downloadSegmentFromPeer$4(RealtimeTableData
Manager.java:666)
     at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.lambda$fetchSegmentToLocal$2(BaseSegmentFetcher.java:127)
     at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java
:50)
     at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:126)
     at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadSegmentFromPeer(Realti
meTableDataManager.java:663)
     at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:606)
     at org.apache.pinot.core.data.manager.realtime.Realtim
eSegmentDataManager.downloadSegmentAndReplace(RealtimeSegmentDataManager.java:1294)
     at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.goOnlineFromConsuming(RealtimeSegmentDataManager.java:1233)

The corresponding code at the commit from which this build was made is shown below. From a operational experience point of view, I think we need the following improvements here:

  1. If the predicate is onlineServers.isEmpty() is true after all attempts, the logs should clearly indicate that this was the reason for the attempt exhaustion.
  2. There should be some way to log the last instance state map seen for this segment during the retries. This can help in knowing the exact EV the Servers were seeing at the time of the failure.
  3. If an exception is thrown in getOnlineServersFromExternalView, that exception should be clearly logged. Maybe this is already happening?

Contributor guide

Tech stack
java
Domain
backenddatabase
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaApache Pinot basicsRetry patterns
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
The issue points to PeerServerSegmentFinder.java, specifically the getPeerServerURIs method which uses a retry policy. The fix should add clear logging when onlineServers is empty after retries, log the last instance state map seen, and ensure exceptions from getOnlineServersFromExternalView are logged. The code is at line 81 of the current version. Also review the RealtimeTableDataManager for context on how this method is called.