Six Stars

tElasticSearchOutput is adding a slash to the document type

I'm trying to write JSON documents to Elasticsearch in a Spark streaming job. I've set the document type to "doc" in the component configuration. I'm getting the following exception:

 

[ERROR]: org.apache.spark.executor.Executor - Exception in task 7.0 in stage 0.0 (TID 7)
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: invalid pattern given doc/
	at org.elasticsearch.hadoop.util.Assert.isTrue(Assert.java:50)
	at org.elasticsearch.hadoop.serialization.field.AbstractIndexExtractor.compile(AbstractIndexExtractor.java:51)
	at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:565)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:58)
	at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
	at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
	at org.apache.spark.scheduler.Task.run(Task.scala:85)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2 REPLIES
Moderator

Re: tElasticSearchOutput is adding a slash to the document type

Hello,

Here exists a jira issue:https://jira.talendforge.org/browse/TBD-3707.

Could you please let us know if it is what you are looking for?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: tElasticSearchOutput is adding a slash to the document type

That seems to be unrelated. I already have my content in a single column of type string:

esSchema.PNG

This Stackoverflow answer makes me think the wrong parameters are being passed in to org.elasticsearch.spark.rdd.EsSpark.saveJsonToEs().  It expects the pattern "indexName/documentType", and it's getting something like "indexName/documentType/" somehow. Here's the relevant configuration for my tElasticSearchOutput:

esConfig.PNG

I tried putting the index name under "Type" and the type ("doc") under "Index" to see if it was just a simple transposition of fields, but that didn't work either.