Four Stars

Validations in Big Data Batch Job

Hi Team,


I have development experience with Talend standard jobs but I am new to developing Big Data Batch Jobs in Talend. I have used tschemacompliance to validate schema data types and lengths in standard jobs in Talend. I am looking for an equivalent component in Big Data Batch job. Can you please help me understand how best we can achieve data type and length validations in a batch job? Looking forward for a timely response. 


Thanks and Regards,

Shiv Soni

Four Stars

Re: Validations in Big Data Batch Job



Can anyone help with this please?


Re: Validations in Big Data Batch Job


The tschemacompliance component is a DI component and will not available in bigdata spark job, so far.

Have you tried to utilize a DI Job to orchestrate your spark job by using tRunJob component?

Best regards




Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.