Elasticsearch & OpenSearch Shard Sizing Calculator: Free Elasticsearch & OpenSearch Shard Sizing Calculator tool: calculate Elasticsearch and OpenSearch primary shard counts, replica storage, cluster disk.
The Architecture of Shards in Distributed Search Engines
In distributed search systems such as Elasticsearch and OpenSearch, an index is a logical namespace subdivided into one or more physical partitions called shards. Each shard is a completely self-contained, fully functional Apache Lucene index, with its own inverted index, term dictionary, stored fields, doc values, and segment files.
Choosing an incorrect shard strategy is one of the most common causes of cluster instability:
- Under-sharding (Shards too large > 50 GB): Massive shards make shard relocation, cluster rebalancing, and node recovery painfully slow. If a node fails, transferring a 100 GB shard across the network can saturate interfaces and trigger cluster timeouts.
- Over-sharding (Too many small shards < 5 GB): Every single shard consumes JVM heap memory to store Lucene segment metadata. Thousands of tiny shards cause massive JVM garbage collection pauses, exhaust thread pools during distributed searches, and waste enormous compute overhead.
The 20 Shards per GB of JVM Heap Rule
Elastic engineering guidelines mandate maintaining no more than 20 to 25 shards per gigabyte of configured JVM heap on each data node:
$text{Max Shards per Node} = text{JVM Heap (GB)} times 20$
For a standard 31 GB JVM Heap node:
$text{Max Shards} = 31 times 20 = mathbf{620text{ shards per node}}$
Exceeding this ratio causes JVM heap starvation, frequent Stop-the-World garbage collection pauses, and cluster-wide red status during query spikes.
Step-by-Step Worked Numerical Sizing Example
Suppose a DevOps team is designing an Elasticsearch logging cluster for Kubernetes microservices:
- Daily Ingestion: 150 GB / day
- Retention Period: 30 days
- Replica Count: 1 replica (1 primary + 1 replica = 2x copies)
- Target Shard Size: 45 GB
- Nodes: 6 dedicated data nodes with 31 GB JVM Heap
- Disk Low Watermark: 85% ($0.85$)
Step 1: Compute Total Primary Storage Volume:
Raw 30-day Ingestion = $150text{ GB/day} times 30text{ days} = 4,500text{ GB}$
Applying 10% Lucene index metadata overhead = $4,500 times 1.10 = mathbf{4,950text{ GB}}; (mathbf{4.95text{ TB}})$
Step 2: Calculate Required Primary and Total Shards:
$text{Primary Shards} = leftlceil frac{4,950text{ GB}}{45text{ GB}} rightrceil = mathbf{110text{ Primary Shards}}$
$text{Total Shards (with 1 Replica)} = 110 times (1 + 1) = mathbf{220text{ Total Shards}}$
Step 3: Evaluate Shard-to-Heap Ratio:
$text{Shards Per Node} = frac{220text{ Shards}}{6text{ Nodes}} approx mathbf{36.7text{ shards per node}}$
Safe Limit = $31text{ GB Heap} times 20 = 620text{ shards per node}$
Heap Shard Capacity Utilized = $frac{36.7}{620} approx mathbf{5.9%}$ (Excellent headroom)
Step 4: Total Physical Disk Sizing across Nodes:
Total Data with Replicas = $4,950text{ GB} times 2 = 9,900text{ GB}; (9.90text{ TB})$
Factoring 85% Disk Watermark = $frac{9,900text{ GB}}{0.85} approx mathbf{11,647text{ GB}}; (mathbf{11.65text{ TB total}})$
Disk Per Data Node = $frac{11.65text{ TB}}{6text{ Nodes}} approx mathbf{1.94text{ TB per node}}$
The 31 GB JVM Heap Limit: Compressed Ordinary Object Pointers
A frequent trap encountered by system administrators is configuring nodes with 64 GB or 128 GB of RAM and attempting to set -Xmx60g for the JVM heap.
When the heap is strictly under 32 GB (typically capped at 31 GB), modern 64-bit JVMs activate Compressed Ordinary Object Pointers (CompressedOops). CompressedOops shifts memory addresses, allowing 64-bit pointers to be stored as 32-bit values. Setting the heap to 32.5 GB disables CompressedOops, forcing all pointers to expand to full 64-bit values. This increases memory overhead by 30% to 40%, leaving less usable heap than a properly configured 31 GB instance!