Five Stars

Unable to publish or fetch data from Kafka

Hi

 

I have build job in Talend integration cloud hybrid edition, create kafka topic and publish data. Below mapping.

I am able to create topic in kafka but when i try to consume data in the terminal, it doesn't show any data.

 

Similar issue with kafka consumer tkafkainput as well.

 

Kafka job.PNG

 

terminal consumer.PNG

 

tkafkainput : always keep running inspite of setting stop after 6000ms.

 tkafkainput.PNG

 

 

Kafka versions in sync.

 

 

 

6 REPLIES
Moderator

Re: Unable to publish or fetch data from Kafka

Hello,

The topic you created in kafka is Ok with you? Could you please indicate on which build version you got this issue?

Here is a jira issue about tKafkaOutput component does not implement "use existing connection"

:https://jira.talendforge.org/browse/TBD-3435

Can you use tkafkaInput(Built-in)-->tlogrow to read your data without using tKafkaConnection component?

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Unable to publish or fetch data from Kafka

Hello,

 

I tried without tkafkaconnection but No luck job keeps running without fetching any data.

 

 

kafkainput wo connection.PNG

 

Talend version: Tested in 2 products

Talend Integration Cloud Hybrid Edition

Version: 6.4.1
Build id: 20170623_1246

----

Talend Open Studio for Big Data

Version: 6.3.1

 

 

 

 

Regards,

M

One Star

Re: Unable to publish or fetch data from Kafka

Hello,

What is the job you are building? Show me how you build specificially

Moderator

Re: Unable to publish or fetch data from Kafka

Hello,

Would you mind posting your tkafkainput component setting screenshot into forum?

The tKafkaInput is a streaming component, even in the DI palette.

Could you please try use tfileoutpudelimited component as output and check the option "use output stream" on the tFileOutputDelimited to correctly put the data into the file?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Unable to publish or fetch data from Kafka

Hello,

 

I am using tkafkaInput and connecting it with tLogRow, when I run this job, messages are read but not printed in tLogRow. Can you please point me the missing setting here.

 I'm using kafka version 0.9.0.1 and Talend Open Studio for Big Data (6.4.1) Free Edition

 

Thanks in advance.

 

 

 

Moderator

Re: Unable to publish or fetch data from Kafka

Hi Shalinigarg,

The tKafkaInput is a streaming component.

If you want to save data to a file with tkafkaInput, you have to check the option "use output stream" on the tFileOutputDelimited to correctly put the data into the file.

Let us know if this solution is OK with you.

Best regards

Sabrina

 

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.