Five Stars

Error handling in Bigdata job

Hi Team,

 

What are the ways to handle errors in Talend BigData Batch job?

 

Some of the bigdata components do not have "OnSubJobError" for eg. tHiveInput. How should we enable error handling in spark?

Tags (2)
5 REPLIES
Moderator

Re: Error handling in Bigdata job

Hello,

The onSubjobError connector is not supported by most Spark components.

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Error handling in Bigdata job

Hi Sabrina,

Then how we should catch errors? Any best practice around that?
Moderator

Re: Error handling in Bigdata job

Hello,

What's kind of error do you want to handle? To capture an exception in Spark jobs?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Error handling in Bigdata job

Hi Sabrina,

 

Lets say, I am performing below operation.

tHiveInput --> tMap (Some Transformation) --> tHiveOutput

 

Even though there is some issue in transformation, YARN application gets completed successfully. I am not able to record/track any issue, unless I go and check the logs.

 

Five Stars

Re: Error handling in Bigdata job

Hi Sabrina,

 

Lets say, I am performing below operation.

tHiveInput --> tMap (Some Transformation) --> tHiveOutput

 

Even though there is some issue in transformation, YARN application gets completed successfully. I am not able to record/track any issue, unless I go and check the logs.

 

Tags (2)