elastic/elasticsearch

During shard relocation some requests might fail to be sent to a shard

Open

#13719 opened on Sep 22, 2015

View on GitHub
 (2 comments) (0 reactions) (0 assignees)Java (76,700 stars) (25,882 forks)batch import
:Distributed/Distributed>bugTeam:Distributedhelp wantedresiliency

Description

When shards relocate then there might be small window in time where requests fail to reach the relocating shard. This happens when a node that lags one cluster state behind has not realized yet that a node has relocated and the relocation source is already removed. Below is a graphical representation of an example course of events. This caused us some trouble in test already because results were unexpected, see https://github.com/elastic/elasticsearch/issues/13266. It affects all actions that inherit from TransportBroadcastAction, TransportBroadcastByNodeAction and might also be problematic for others. For example: an optimize request might never reach a shard if it is relocating, indices stats may report wrong statistics, see https://github.com/elastic/elasticsearch/issues/13266#issuecomment-138470051, etc.

We should check if we can get away with just sending requests to relocation targets too for the affected actions or if we need to implement these kind of requests as replication action like we did for refresh and flush.

recovery-issues

finally

I: shard is relocating from n2 to n3 II: CS2 signals that n3 has started its shard and n2 can remove its own copy. shard on n2 is therefore closed. But n1 lags behind one cluster state and still expects an up and running primary on n2 III: if now n1 sends an optimize, indices stats request or the likes, it will send the request to n2 (based on CS1) but that does not not have the shard anymore.

Contributor guide

Tech stack
java
Domain
backenddatabase
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
5
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Elasticsearch cluster stateshard relocationTransportBroadcastAction
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
10
Research direction
Investigate the two approaches suggested: either send requests to relocation targets for affected actions (like optimize, indices stats) or implement these requests as replication actions similar to refresh and flush. Examine the current implementations of TransportBroadcastAction and TransportBroadcastByNodeAction in the Elasticsearch codebase. Review the linked issues (#13266 and comments) for additional context and any partial solutions. Consider whether the problem still exists in the latest version or if it has been addressed by other changes.