Problem when using different Kafka API versions in standard and Big Data Streaming Jobs

Problem Description

You are testing a recommendation engine, and you want to use the tKafkaInput and tKafkaOutput components to push data to and retrieve data from Kafka. You are using Confluent Kafka 1.0, so you select version 0.10 of the tKafka* components.

 

Both components work well in standard Jobs, but Big Data Streaming Jobs fail to retrieve the data.

 

Root Cause

The standard Job is calling kafka_2.10-0.10.0.1.jar, and is working properly. But the Spark Streaming Job is using spark-streaming-kafka-assembly_2.10-1.6.0-hadoop2.6.0.jar, which is compatible with Kafka 0.8.2.1, not with Kafka 1.0.

 

Solution

Ensure that both of your Jobs are using the version of the JAR file that matches the version of Spark you're using.

 

For more information, see the Kafka Integration Guide and the Spark Streaming Programming Guide.

Version history
Revision #:
10 of 10
Last update:
‎03-25-2019 06:31 AM
Updated by: