opensearch-project/OpenSearch

[BUG] Unable to enable remote state repository independently of data repositories

Open

#13,523 创建于 2024年5月3日

在 GitHub 查看
 (4 评论) (0 反应) (1 负责人)Java (1,505 fork)batch import
StorageStorage:Remotebuggood first issuelow hanging fruitv2.15.0

仓库指标

Star
 (8,123 star)
PR 合并指标
 (平均合并 5天 9小时) (30 天内合并 266 个 PR)

描述

Describe the bug

Enabling just remote state repo (support added via PR #11858 ) starts a remote store migration. This migration doesn’t go through. The shards stay unassigned.

Related component

Storage

To Reproduce

  1. Launch a docrep cluster with remote state enabled. Do not enable remote segment and remote translog repositories.
  2. Create an index.
  3. Shard creation fails and shards stay unassigned.

Expected behavior

Enabling only remote state repository should not start a migration to remote store

Additional Details

Exception stack trace

[2024-05-02T13:17:25,436][INFO ][o.o.i.IndexService       ] [node-1] [idx1] DocRep shard [idx1][3] is migrating to remote
[2024-05-02T13:17:25,436][WARN ][o.o.i.c.IndicesClusterStateService] [node-1] [idx1][3] marking and sending shard failed due to [failed to create shard]
java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
    at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?]
    at org.opensearch.repositories.RepositoriesService.repository(RepositoriesService.java:568) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory.newDirectory(RemoteSegmentStoreDirectoryFactory.java:61) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.index.IndexService.createShard(IndexService.java:512) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.indices.IndicesService.createShard(IndicesService.java:1025) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.indices.IndicesService.createShard(IndicesService.java:213) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:672) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:649) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:294) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:608) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:595) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:563) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:486) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:188) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
    at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Proposed solution The migration flow uses the isRemoteStoreNode( ) method. This method checks for the presence of any remote_store node attribute on the node: https://github.com/opensearch-project/OpenSearch/blob/ed33488aa426bd618685729fc638adad763f6ff7/server/src/main/java/org/opensearch/cluster/node/DiscoveryNode.java#L464-L471

Given that the state repo should be independent now, we should have distinct methods to identify whether the cluster state or data is remote-backed.

贡献者指南