[BUG] cluster.max_shards_per_node incorrectly counts warm nodes in data node count

Start in server/src/main/java/org/opensearch/indices/ShardLimitValidator.java at checkShardLimit(), then inspect DiscoveryNode.isWarmNode() and DiscoveryNodeRole.WARM_ROLE. Reproduce with the described hot-and-warm cluster and verify that warm nodes are excluded from the count and index creation fails at the hot-node shard limit.

Describe the bug

Description

The cluster.max_shards_per_node setting computes the cluster-wide shard creation limit as:

max_shards_per_node × data_node_count

However, data_node_count includes all nodes where canContainData = true, which includes warm nodes. This inflates the allowed shard ceiling beyond what hot nodes alone can support.

Impact

In a cluster with 6 hot data nodes and 8 warm nodes, with the default cluster.max_shards_per_node = 1000:

• Current behavior: Limit is 1000 × 14 = 14,000 shards
• Expected behavior: Limit should be 1000 × 6 = 6,000 shards (hot nodes only)

This allows hot-tier index creation far beyond what the hot nodes can support, since warm nodes are not intended to host hot indices. It also creates inconsistency with external tooling (e.g., Blue/Green pre-validation) that correctly scopes the shard limit to hot nodes.

Root Cause

In ShardLimitValidator.checkShardLimit():

int nodeCount = state.getNodes().getDataNodes().size();

getDataNodes() returns all nodes where isDataNode() returns true. isDataNode() is implemented as:

public boolean isDataNode() {
    return roles.stream().anyMatch(DiscoveryNodeRole::canContainData);
}

Both DATA_ROLE and WARM_ROLE are defined with canContainData = true, so warm nodes are counted.

Steps to Reproduce

1. Set up an OpenSearch cluster with hot data nodes and warm nodes (e.g., 6 hot + 8 warm)
2. Set cluster.max_shards_per_node = 1000 (default)
3. Create indices up to the expected hot-only limit (6,000 shards)
4. Observe that additional index creation continues to succeed, up to the inflated limit of 14,000 shards

Proposed Fix

Exclude warm nodes from the data node count in ShardLimitValidator.checkShardLimit():

int nodeCount = (int) state.getNodes().getDataNodes().values().stream()
    .filter(node -> !node.isWarmNode())
    .count();

DiscoveryNode.isWarmNode() is an existing method that checks roles.contains(DiscoveryNodeRole.WARM_ROLE). This uses existing APIs with no new dependencies.

Expected Behavior

cluster.max_shards_per_node should reflect the capacity of hot data nodes only, since it is intended to protect hot-tier index creation.

OpenSearch Version

Observed in: 2.19.0

The warm role was introduced in OpenSearch 2.4. This bug affects all versions 2.4 and later that use ShardLimitValidator in clusters with warm nodes configured.

Related

• ShardLimitValidator.java
• DiscoveryNode.isWarmNode()
• DiscoveryNodeRole.WARM_ROLE

Related component

Cluster Manager

To Reproduce

To Reproduce

1. Set up an OpenSearch cluster (version 2.4+) with both hot and warm data nodes. For example:

   • 2 hot data nodes: node.roles: [data, ingest, cluster_manager]
   • 3 warm nodes: node.roles: [warm]

2. Keep cluster.max_shards_per_node at its default value of 1000, or set it explicitly:

   curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '
   {
     "persistent": {
       "cluster.max_shards_per_node": 1000
     }
   }'
   

3. Verify the node count:

   curl "localhost:9200/_cat/nodes?v&h=ip,node.role"
   

   Expected: 5 data-capable nodes (2 hot + 3 warm).

4. Create hot-tier indices until the expected hot-only limit is reached. With 2 hot nodes, the expected limit is 2 × 1000 = 2000 shards:

   for i in $(seq 1 400); do
     curl -X PUT "localhost:9200/test-index-$i" -H 'Content-Type: application/json' -d '
     {
       "settings": {
         "number_of_shards": 5,
         "number_of_replicas": 0
       }
     }'
   done
   

   This creates 400 × 5 = 2000 shards.

5. Attempt to create one more index beyond the hot-only limit:

   curl -X PUT "localhost:9200/test-index-401"
   

   Observed: The request succeeds. Index creation continues to succeed until ~5000 shards, because OpenSearch computes the cluster-wide limit as 1000 × 5 = 5000 (counting all 5 data-capable nodes including warm).

   Expected: The request should fail with a validation_exception:

   this action would add [5] total shards, but this cluster currently has [2000]/[2000] maximum shards open
   

Expected behavior

Expected Behavior

cluster.max_shards_per_node should only consider hot data nodes when computing the cluster-wide shard limit. Warm nodes should be excluded because:

• Warm nodes are not intended to host hot-tier (regular) indices
• The setting is meant to protect hot nodes from over-allocation
• Counting warm nodes inflates the effective limit, allowing hot shard creation far beyond what the hot tier can safely support
• Tier-aware pre-validation tooling (e.g., in managed services) uses the correct hot-only count, creating inconsistency with OpenSearch's runtime behavior

For the example above, the expected computed limit should be 1000 × 2 = 2000 shards, not 1000 × 5 = 5000.

Additional Details

No response