You are testing a recommendation engine, and you want to use the tKafkaInput and tKafkaOutput components to push data to and retrieve data from Kafka. You are using Confluent Kafka 1.0, so you select version 0.10 of the tKafka* components.
Both components work well in standard Jobs, but Big Data Streaming Jobs fail to retrieve the data.
The standard Job is calling kafka_2.10-0.10.0.1.jar, and is working properly. But the Spark Streaming Job is using spark-streaming-kafka-assembly_2.10-1.6.0-hadoop2.6.0.jar, which is compatible with Kafka 0.8.2.1, not with Kafka 1.0.
Ensure that both of your Jobs are using the version of the JAR file that matches the version of Spark you're using.