ElasticSearch status in red after full restart vIDM cluster

After completely shutting down a vIDM cluster and starting it up again, I noticed that the ElasticSearch status on all three nodes was in red. You can see this in the System Diagnosis Dashboard.

Initially I thought this was just a matter of time before everything was properly synced. But unfortunately the status remained red.

In the VMware documentation you can find information on how to troubleshoot ElasticSeearch. You can find it here.

To check the health state via command line, open an SSH session to one of the vIDM nodes and enter this command:

curl ‘http://localhost:9200/_cluster/health?pretty’

which gives the following output:

curl ‘http://localhost:9200/_cluster/health?pretty’
{
“cluster_name” : “horizon”,
“status” : “red”,
“timed_out” : false,
“number_of_nodes” : 3,
“number_of_data_nodes” : 3,
“active_primary_shards” : 28,
“active_shards” : 56,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 14,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 80.0
}

A restart of the ElasticSearch service did not change anything. Also, in the logs no errors could be found.

Looking more closely at the output of health check command however, I could see that there were unassigned shards.

“unassigned_shards” : 14,

To understand the meaning of shards, please see: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/_basic_concepts.html

To get more detailed info on the shards, execute this command on one of the vIDM nodes:

url ‘localhost:9200/_cat/shards?v’

From the output of this command I could see that shards after the startup of the cluster were assigned successfully and that the unassigned were only from the moment the cluster was shutdown.

This indicated that the ElasticSearch cluster was running without any errors, but there were some older entries stuck in unassigned mode.

The table below shows the output of the unassigned shards:

curl ‘localhost:9200/_cat/shards?v’ | grep UNASS

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5822 100 5822 0 0 844k 0 –:–:– –:–:– –:–:– 947k
searchentities 0 p UNASSIGNED
searchentities 0 r UNASSIGNED

(There were 14 unassigned shards, but for clarity I removed the others.)

To resolve this, you have to manually reallocate the shards.

Use the following command for this:

curl -XPOST ‘localhost:9200/_cluster/reroute’ -d ‘{
“commands”: [{
“allocate”: {
“index”: “one_of_the_indexes“,
“shard”: shard_number,
“node”: “one_of_nodes“,
“allow_primary”: 1
}
}]
}’

To find the name of the nodes, execute this command:

curl ‘localhost:9200/_cat/nodes?v’

The name of the nodes is shown in the last column:

In my case the command to reallocate the shards looks like this:

curl -XPOST ‘localhost:9200/_cluster/reroute’ -d ‘{“commands”:[{“allocate”:{“index”: “searchentities”,”shard”: 0,”node”: “Ringleader”,”allow_primary”: 1}}]}’

Listing the unassigned shards again, you can now see that “searchentities” dissappeared from the unassigned list.

curl ‘localhost:9200/_cat/shards?v’ | grep UNASS

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5822 100 5822 0 0 844k 0 –:–:– –:–:– –:–:– 947k


and the searchentities shards have been added to the assigned:

curl ‘localhost:9200/_cat/shards?v’ | grep “searchentities”
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2240 100 2240 0 0 286k 0 –:–:– –:–:– –:–:– 364k
searchentities 0 p STARTED
searchentities 0 r STARTED

If you have multiple unassigned shards, you should run the command to reallocate the shards for each of them.

When all shards have been reallocated, the ElasticSearch cluster becomes green again 🙂

Please follow and like us:
error

Leave a Reply

Your email address will not be published. Required fields are marked *