Elasticsearch can throw disk usage exceptions like these:
[INFO ][cluster.allocation] [NodeB] low disk usage [90%] exceeded on [ NodeA] free: 5gb[10%], replicas will not be assigned to this node
[WARN ][cluster.allocation] [NodeA] high disk usage [95%] exceeded on [ myELK -Node2] free: 4.8gb[9.9%], shards will be relocated away from this node
[INFO ][cluster.allocation] [NodeA] high disk usage exceeded on one or more nodes, rerouting shards....
This can cause a drastic performance reduction of the Elasticsearch service, and finally lead to its crash.
A huge number of logs can be written to the Elasticsearch service periodically. If you don't have a proper archival process in place, data in the Elasticsearch cluster will grow uncontrollably, which can lead to the loss of valuable log data if you don't provide enough disk space.
Curator is a tool from Elastic (the company behind Elasticsearch) to help manage your Elasticsearch cluster. Curator is a Python-based tool that can help you manage Elasticsearch indices. Deleting old indices is one of the primary use cases for Curator.
As a prerequisite, you must install Python version 3.4+. You can then install Curator using the following pip command:
pip install elasticsearch-curator
This should install Curator in the machine.
Additional commands to get started with Curator
List the available indices in the ElasticSearch cluster
curator show indices --all-indices
Remove old indices
It is better to do a dry run before initiating an automation script to delete data. Curator provides a dry-run flag to output the verbose data from a test execution without actually deleting the indices:
curator --dry-run --host <ip address> delete indices --time-unit days --older-than 60 --timestring '%Y%m%d'
Once the test runs are verified, you can automate the purging of old indices using scripts (for example using a cron job) as shown below:
curator --host <ip address> delete indices --time-unit days --older-than 60 --timestring '%Y%m%d'
This will delete the indices older than 60 days.
YAML-based Curator configuration
A more elegant way to configure and automate Curator execution is using a YAML configuration. This needs 2 configuration files, namely:
Contains configuration details such as the cluster location and port settings. Sample file content is shown below:
Configures an action list to be executed by Curator. For example, to back up and purge indices of data from logstash, with the prefix logstash, use the following configuration:
Delete indices older than 30 days (based on index name).
— filtertype: pattern
— filtertype: age
Once the two YAML files are configured, you can initiate a dry run of Curator execution with following command:
curator ./curator-action.yml --config ./curator-config.yml --dry-run
2018–02–05 11:30:35,075 INFO Preparing Action ID: 1, "delete_indices"
2018–02–05 11:30:35,095 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 30 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2018–02–05 11:30:35,095 INFO DRY-RUN MODE. No changes will be made.
2018–02–05 11:30:35,095 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2018–02–05 11:30:35,095 INFO Action ID: 1, "delete_indices" completed.
2018–02–05 11:30:35,095 INFO Job completed.
The --dry-run mode will not actually delete the index. It can be used to verify the output of the action.
Once the dry run verification is completed, you can schedule the actual run in a cron, using crontab -e as shown below:
00 6 * * * root curator /path/curator-action.yml --config /path/curator-config.yml
This configuration will clean up the indices older than 30 days, running every day at 6 AM.
Execution verbose is:
2018–02–05 12:26:40,525 INFO Action #1: delete_indices
2018–02–05 12:26:45,669 INFO Deleting selected indices: [u’logstash-2018–01–03']
2018–02–05 12:26:45,789 INFO — -deleting index logstash-2018–01–03
2018–02–05 12:26:45,960 INFO Action #1: completed
2018–02–05 12:27:18,961 INFO Job completed.
... View more
Hi, So you could do the following to get handle on JMS destination 1. In a Non clustered environment, the format of the string to use is: <jmsservername>/<jmsmodulename>!<destinationname> 2. In a clustered environment use: <jmsmodulename>!<destinationname>
... View more
Hi Eric, The TDM bean access is working as expected also on 6.4.1. There was some initial GUI error on my 6.4.1 Studio causing the screen freeze and not allowing class path edit. I had a fresh 6.4.1 installation with new project workspace, and the custom code jar can be added to the TDM class path via new Database option. But your solution/approach is neater. Thanks/Regards
... View more
- Works fine on 6.3.1: The java/bean usage is as follows. Create a java class/bean within Talend Studio's Integration perspective > Code/beans. Then switch to java perspective (Package explorer) , right click on the Project/code and export as Jar. In this case, it will list some pom*.xml files within Code folder for export. Select all and then create the jar ignoring some exception/error pop-up. Then switch to Mapping perspective , new > Database , edit class path and add the jar. Next in a Map, one can access the routine by using Java function passing the packagename and class name (for example beans.MyClass) followed by routine(and it's arguments). .This works fine in 6.3.1. - How is this done on 6.4.1 and higher versions ?! We have noticed that in Studio 6.4.1, it doesn't show any option to import the jar file onto TDM class path(attached screenshot). Ideally (could be a feature request) it would have been great if we could access the bean class directly without having to import any jar files.
... View more
Hi Richard, Thanks for contacting us via OTRS Ticketing system. We'v raised this issue with the R&D. There is a possible problem with the xmlmapping component in the DataService which is not able to process the namespace correctly. I'v also noticed that this issue is not reproducible in the latest TESB5.1.1. In the meantime, is it possible for you to apply the workaround on modifying the .net request to place the namespace in the main envelope tag. Thanks/Regards
... View more