[BUG] Unable to enable remote state repository independently of data repositories
#13.523 geöffnet am 3. Mai 2024
Repository-Metriken
- Stars
- (8.123 Stars)
- PR-Merge-Metriken
- (Durchschn. Merge 5T 9h) (266 gemergte PRs in 30 T)
Beschreibung
Describe the bug
Enabling just remote state repo (support added via PR #11858 ) starts a remote store migration. This migration doesn’t go through. The shards stay unassigned.
Related component
Storage
To Reproduce
- Launch a docrep cluster with remote state enabled. Do not enable remote segment and remote translog repositories.
- Create an index.
- Shard creation fails and shards stay unassigned.
Expected behavior
Enabling only remote state repository should not start a migration to remote store
Additional Details
Exception stack trace
[2024-05-02T13:17:25,436][INFO ][o.o.i.IndexService ] [node-1] [idx1] DocRep shard [idx1][3] is migrating to remote
[2024-05-02T13:17:25,436][WARN ][o.o.i.c.IndicesClusterStateService] [node-1] [idx1][3] marking and sending shard failed due to [failed to create shard]
java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?]
at org.opensearch.repositories.RepositoriesService.repository(RepositoriesService.java:568) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory.newDirectory(RemoteSegmentStoreDirectoryFactory.java:61) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.index.IndexService.createShard(IndexService.java:512) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.indices.IndicesService.createShard(IndicesService.java:1025) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.indices.IndicesService.createShard(IndicesService.java:213) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:672) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:649) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:294) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:608) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:595) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:563) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:486) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:188) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Proposed solution
The migration flow uses the isRemoteStoreNode( ) method. This method checks for the presence of any remote_store node attribute on the node:
https://github.com/opensearch-project/OpenSearch/blob/ed33488aa426bd618685729fc638adad763f6ff7/server/src/main/java/org/opensearch/cluster/node/DiscoveryNode.java#L464-L471
Given that the state repo should be independent now, we should have distinct methods to identify whether the cluster state or data is remote-backed.