Talend 6.2 and Elasticsearch 2.3.5

Highlighted
One Star

Talend 6.2 and Elasticsearch 2.3.5

Any thoughts on how to go about loading data into ES2.3.5 using Talend 6.2 ?
I tried a few components and not much luck. My first attempt was with the tElasticsearchindex component.  This gives a ton of errors which point to issues with the plugin or the base libraries in use.
connecting to socket on port 3369
connected
Exception in thread "main" java.lang.Error: Unresolved compilation problems: 
org.elasticsearch.common.settings.ImmutableSettings cannot be resolved to a type
The constructor TransportClient(Settings) is undefined
The constructor InetSocketTransportAddress(String, Integer) is undefined
at local_project.elasticsearch_data_ingest_0_1.ElasticSearch_Data_Ingest.tFileInputDelimited_1Process(ElasticSearch_Data_Ingest.java:495)
at local_project.elasticsearch_data_ingest_0_1.ElasticSearch_Data_Ingest.runJobInTOS(ElasticSearch_Data_Ingest.java:978)
at local_project.elasticsearch_data_ingest_0_1.ElasticSearch_Data_Ingest.main(ElasticSearch_Data_Ingest.java:835)
Next I tried utilizing the tREST component, but directly posting a JSON structure to the body.  Utilizing port 9300 (http port for ES), I still get its an invalid HTTP response error.  
The next one I tried is utilizing the tHTTPRequest and also tRESTClient. Not much luck.
Any thoughts or pointers to get going with a load to Elasticsearch 2.3.5 ?
TIA
Ben
Moderator

Re: Talend 6.2 and Elasticsearch 2.3.5

Hi,
Have you tried to use tElasticSearchXXX components to achieve your goal?
TalendHelpCenter:tElasticSearchInput 
TalendHelpCenter:tElasticSearchOutput
Note: These tElasticSearchXXX components will be available in the Palette of Talend Studio on the condition that you have subscribed to one of the Talend solutions with Big Data.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Talend 6.3 and Elasticsearch

Hi Team,

 

i have created one simple big data job from json file to elastic search. i am new to the talend. my job is failing with the below errors. can some help me how to resolve this issue.

WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).

[Stage 0Smiley Embarassed (0 + 3) / 3]
[ERROR]: org.apache.spark.executor.Executor - Exception in task 2.0 in stage 0.0 (TID 2)
java.io.IOException: Failed to connect to /192.168.0.5:65100
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:358)
at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324)
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
[Stage 0Smiley Embarassed (0 + 3) / 3]
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:480)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection timed out: no further information: /192.168.0.5:65100
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
[Stage 0Smiley Embarassed (0 + 2) / 3]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
... 1 more
[WARN ]: org.apache.spark.scheduler.TaskSetManager - Lost task 2.0 in stage 0.0 (TID 2, localhost): java.io.IOException: Failed to connect to /192.168.0.5:65100
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
at org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:358)
at org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324)
Job test_elastic ended at 21:30 30/01/2019. [exit code=1]
[Stage 0Smiley Embarassed (0 + 2) / 3]
at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:633)
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:459)
at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:488)
at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:480)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:480)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:252)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection timed out: no further information: /192.168.0.5:65100
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)

 

Thanks

 

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.