Six Stars

tElasticSearchOutput is adding a slash to the document type

I'm trying to write JSON documents to Elasticsearch in a Spark streaming job. I've set the document type to "doc" in the component configuration. I'm getting the following exception:


[ERROR]: org.apache.spark.executor.Executor - Exception in task 7.0 in stage 0.0 (TID 7)
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: invalid pattern given doc/
	at org.elasticsearch.hadoop.util.Assert.isTrue(
	at org.elasticsearch.hadoop.serialization.field.AbstractIndexExtractor.compile(
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:58)
	at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
	at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
	at org.apache.spark.executor.Executor$
	at java.util.concurrent.ThreadPoolExecutor.runWorker(
	at java.util.concurrent.ThreadPoolExecutor$

Re: tElasticSearchOutput is adding a slash to the document type


Here exists a jira issue:

Could you please let us know if it is what you are looking for?

Best regards


Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: tElasticSearchOutput is adding a slash to the document type

That seems to be unrelated. I already have my content in a single column of type string:


This Stackoverflow answer makes me think the wrong parameters are being passed in to org.elasticsearch.spark.rdd.EsSpark.saveJsonToEs().  It expects the pattern "indexName/documentType", and it's getting something like "indexName/documentType/" somehow. Here's the relevant configuration for my tElasticSearchOutput:


I tried putting the index name under "Type" and the type ("doc") under "Index" to see if it was just a simple transposition of fields, but that didn't work either.