Deleting old indices in Elasticsearch using Curator

Problem Description

Elasticsearch can throw disk usage exceptions like these:

[INFO ][cluster.allocation] [NodeB] low disk usage [90%] exceeded on [ NodeA] free: 5gb[10%], replicas will not be assigned to this node
[WARN ][cluster.allocation] [NodeA] high disk usage [95%] exceeded on [ myELK -Node2] free: 4.8gb[9.9%], shards will be relocated away from this node
[INFO ][cluster.allocation] [NodeA] high disk usage exceeded on one or more nodes, rerouting shards....

 

This can cause a drastic performance reduction of the Elasticsearch service, and finally lead to its crash.

 

Root Cause

A huge number of logs can be written to the Elasticsearch service periodically. If you don't have a proper archival process in place, data in the Elasticsearch cluster will grow uncontrollably, which can lead to the loss of valuable log data if you don't provide enough disk space.

 

Solution

Curator is a tool from Elastic (the company behind Elasticsearch) to help manage your Elasticsearch cluster. Curator is a Python-based tool that can help you manage Elasticsearch indices. Deleting old indices is one of the primary use cases for Curator.

 

As a prerequisite, you must install Python version 3.4+. You can then install Curator using the following pip command:

pip install elasticsearch-curator

 

This should install Curator in the machine.

 

Additional commands to get started with Curator

List the available indices in the ElasticSearch cluster

curator show indices --all-indices

 

Remove old indices

It is better to do a dry run before initiating an automation script to delete data. Curator provides a dry-run flag to output the verbose data from a test execution without actually deleting the indices:

curator --dry-run --host <ip address> delete indices --time-unit days --older-than 60 --timestring '%Y%m%d'

 

Once the test runs are verified, you can automate the purging of old indices using scripts (for example using a cron job) as shown below:

curator --host <ip address> delete indices --time-unit days --older-than 60 --timestring '%Y%m%d'

This will delete the indices older than 60 days.

 

YAML-based Curator configuration

A more elegant way to configure and automate Curator execution is using a YAML configuration. This needs 2 configuration files, namely:

  • curator-config.yml

    Contains configuration details such as the cluster location and port settings. Sample file content is shown below:

    client:
      hosts:
        — localhost
      port: 9200
      url_prefix:
      use_ssl: False
      certificate:
      client_cert:
      client_key:
      aws_key:
      aws_secret_key:
      aws_region:
      ssl_no_validate: False
      http_auth:
      timeout: 100
      master_only: False
    logging:
      loglevel: INFO
      logfile:
      logformat: default
      blacklist: [‘elasticsearch’]

     

  • curator-action.yml

    Configures an action list to be executed by Curator. For example, to back up and purge indices of data from logstash, with the prefix logstash, use the following configuration:

    actions:
     
      1:
        action: delete_indices
        description: >-
          Delete indices older than 30 days (based on index name).
        options:
          ignore_empty_list: True
          timeout_override:
          continue_if_exception: False
          disable_action: False
        filters:
        — filtertype: pattern
          kind: prefix
          value: logstash-
          exclude:
        — filtertype: age
          source: name
          direction: older
          timestring: ‘%Y.%m.%d’
          unit: days
          unit_count: 30
          exclude:

 

Once the two YAML files are configured, you can initiate a dry run of Curator execution with following command:

curator ./curator-action.yml --config ./curator-config.yml --dry-run

2018–02–05 11:30:35,075 INFO      Preparing Action ID: 1, "delete_indices"
2018–02–05 11:30:35,095 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 30 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2018–02–05 11:30:35,095 INFO      DRY-RUN MODE.  No changes will be made.
2018–02–05 11:30:35,095 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2018–02–05 11:30:35,095 INFO      Action ID: 1, "delete_indices" completed.
2018–02–05 11:30:35,095 INFO      Job completed.

The --dry-run mode will not actually delete the index. It can be used to verify the output of the action.

 

Once the dry run verification is completed, you can schedule the actual run in a cron, using crontab -e as shown below:

00 6 * * * root curator /path/curator-action.yml --config /path/curator-config.yml

This configuration will clean up the indices older than 30 days, running every day at 6 AM.

 

Execution verbose is:

2018–02–05 12:26:40,525 INFO Action #1: delete_indices
2018–02–05 12:26:45,669 INFO Deleting selected indices: [u’logstash-2018–01–03']
2018–02–05 12:26:45,789 INFO — -deleting index logstash-2018–01–03
2018–02–05 12:26:45,960 INFO Action #1: completed
2018–02–05 12:27:18,961 INFO Job completed.
Version history
Revision #:
15 of 15
Last update:
‎09-29-2018 12:13 AM
Updated by: