Log Server does not start up

Talend Version 6.3.1

Summary

Log Server fails to start and throws an exception in the logs.
Additional Versions Applicable to other versions
Product Log Server
Component  
 

The Log Server fails to start, with the following warning message:

[2017-07-25 10:00:29,706][WARN ][indices.cluster ] [[logstash-2017.03.14][2]] marking and sending shard failed due to [failed recovery]
[logstash-2017.03.14][[logstash-2017.03.14][2]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to open reader on writer]; nested: FileSystemException[/app/talend/logserv/elasticsearch-2.4.0/data/elasticsearch/nodes/0/indices/logstash-2017.03.14/2/index/_r.cfe: Too many open files];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: [logstash-2017.03.14][[logstash-2017.03.14][2]] EngineCreationFailureException[failed to open reader on writer]; nested: FileSystemException[/app/talend/logserv/elasticsearch-2.4.0/data/elasticsearch/nodes/0/indices/logstash-2017.03.14/2/index/_r.cfe: Too many open files];
at org.elasticsearch.index.engine.InternalEngine.createSearcherManager(InternalEngine.java:292)
at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:163)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
... 5 more
Caused by: java.nio.file.FileSystemException: /app/talend/logserv/elasticsearch-2.4.0/data/elasticsearch/nodes/0/indices/logstash-2017.03.14/2/index/_r.cfe: Too many open files
Problem root cause According to the warning message, the number of open files configured on the system is too low.
Solution or Workaround
  1. Check the number of open files for a Linux-based operating system, by using the command below:

    ulimit -Ha

    The limit of open files can be increased. For example, 65536 is a comfortable value for the open file limit.

  2. Increase the value, following the instructions here: Linux Increase The Maximum Number Of Open Files / File Descriptors (FD).

    Note: Talend recommends that UNIX admins undertake this activity.

  3. Restart the services.
JIRA ticket number  
Version history
Revision #:
4 of 4
Last update:
‎09-29-2018 12:13 AM
Updated by:
 
Labels (3)