tkafkaInput Advanced settings and error handling

Six Stars

tkafkaInput Advanced settings and error handling

Hello,

 

I am currently working with the tkafka components and for the moment  all is working on a success scenario, but i am facing lot of issues on the error handling.  For example the advanced settings, i gave ack=all, retries=10, but none of these seems to get considered. I tried to test a scenario where i brought down a broker with ack=all, but the kafka input doesn't try to turn down that transaction rather continued to process the next message and finished with success, i had message lost during this transaction.

 

Is this a bug ? or there are other ways of configuring these additional properties ?

 

Would be really helpful with some pointers on this topic.

 

Thanks

Moderator

Re: tkafkaInput Advanced settings and error handling

Hello,

For further information about the consumer properties you can define in Kafka consumer properties, please see the section describing the consumer configuration in Kafka's documentation in http://kafka.apache.org/documentation.html#consumerconfigs.

Best regards

Sabrina

 
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: tkafkaInput Advanced settings and error handling

Hi,

 

Am getting the following error "Exception in component tKafkaInput_1
kafka.common.MessageSizeTooLargeException: Found a message larger than the maximum fetch size of this consumer on topic "

 

I have increased my fetch size as attached . Please assist .

 

Tags (1)
Six Stars

Re: tkafkaInput Advanced settings and error handling

I see the properties being defined in the kafka node, are they inline with
the properties defined in the kafka server ? May be that could be a point
to look
Moderator

Re: tkafkaInput Advanced settings and error handling

Hello @jibe

We see that this topic is set as resolved.

https://community.talend.com/t5/Design-and-Development/tkafkaInput-Advanced-settings/m-p/142543

Is this solution "clear your additional Kafka properties as in your attached config and  try using the "fetch.message.max.bytes" parameter" Ok with you?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Moderator

Re: tkafkaInput Advanced settings and error handling

Hello @sensiva

What's kafka API version are you using? The kafka API changed between 0.8 and 0.9.

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch