Elasticsearch can throw disk usage exceptions like these:
[INFO ][cluster.allocation] [NodeB] low disk usage [90%] exceeded on [ NodeA] free: 5gb[10%], replicas will not be assigned to this node [WARN ][cluster.allocation] [NodeA] high disk usage [95%] exceeded on [ myELK -Node2] free: 4.8gb[9.9%], shards will be relocated away from this node [INFO ][cluster.allocation] [NodeA] high disk usage exceeded on one or more nodes, rerouting shards....
This can cause a drastic performance reduction of the Elasticsearch service, and finally lead to its crash.
A huge number of logs can be written to the Elasticsearch service periodically. If you don't have a proper archival process in place, data in the Elasticsearch cluster will grow uncontrollably, which can lead to the loss of valuable log data if you don't provide enough disk space.
Curator is a tool from Elastic (the company behind Elasticsearch) to help manage your Elasticsearch cluster. Curator is a Python-based tool that can help you manage Elasticsearch indices. Deleting old indices is one of the primary use cases for Curator.
As a prerequisite, you must install Python version 3.4+. You can then install Curator using the following pip command:
pip install elasticsearch-curator
This should install Curator in the machine.
curator show indices --all-indices
It is better to do a dry run before initiating an automation script to delete data. Curator provides a dry-run flag to output the verbose data from a test execution without actually deleting the indices:
curator --dry-run --host <ip address> delete indices --time-unit days --older-than 60 --timestring '%Y%m%d'
Once the test runs are verified, you can automate the purging of old indices using scripts (for example using a cron job) as shown below:
curator --host <ip address> delete indices --time-unit days --older-than 60 --timestring '%Y%m%d'
This will delete the indices older than 60 days.
A more elegant way to configure and automate Curator execution is using a YAML configuration. This needs 2 configuration files, namely:
Contains configuration details such as the cluster location and port settings. Sample file content is shown below:
client: hosts: — localhost port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: aws_key: aws_secret_key: aws_region: ssl_no_validate: False http_auth: timeout: 100 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: [‘elasticsearch’]
Configures an action list to be executed by Curator. For example, to back up and purge indices of data from logstash, with the prefix logstash, use the following configuration:
actions: 1: action: delete_indices description: >- Delete indices older than 30 days (based on index name). options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: — filtertype: pattern kind: prefix value: logstash- exclude: — filtertype: age source: name direction: older timestring: ‘%Y.%m.%d’ unit: days unit_count: 30 exclude:
Once the two YAML files are configured, you can initiate a dry run of Curator execution with following command:
curator ./curator-action.yml --config ./curator-config.yml --dry-run 2018–02–05 11:30:35,075 INFO Preparing Action ID: 1, "delete_indices" 2018–02–05 11:30:35,095 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 30 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. 2018–02–05 11:30:35,095 INFO DRY-RUN MODE. No changes will be made. 2018–02–05 11:30:35,095 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices". 2018–02–05 11:30:35,095 INFO Action ID: 1, "delete_indices" completed. 2018–02–05 11:30:35,095 INFO Job completed.
The --dry-run mode will not actually delete the index. It can be used to verify the output of the action.
Once the dry run verification is completed, you can schedule the actual run in a cron, using crontab -e as shown below:
00 6 * * * root curator /path/curator-action.yml --config /path/curator-config.yml
This configuration will clean up the indices older than 30 days, running every day at 6 AM.
Execution verbose is:
2018–02–05 12:26:40,525 INFO Action #1: delete_indices 2018–02–05 12:26:45,669 INFO Deleting selected indices: [u’logstash-2018–01–03'] 2018–02–05 12:26:45,789 INFO — -deleting index logstash-2018–01–03 2018–02–05 12:26:45,960 INFO Action #1: completed 2018–02–05 12:27:18,961 INFO Job completed.