Validations in Big Data Batch Job

Four Stars

Validations in Big Data Batch Job

Hi Team,


I have development experience with Talend standard jobs but I am new to developing Big Data Batch Jobs in Talend. I have used tschemacompliance to validate schema data types and lengths in standard jobs in Talend. I am looking for an equivalent component in Big Data Batch job. Can you please help me understand how best we can achieve data type and length validations in a batch job? Looking forward for a timely response. 


Thanks and Regards,

Shiv Soni

Four Stars

Re: Validations in Big Data Batch Job



Can anyone help with this please?


Re: Validations in Big Data Batch Job


The tschemacompliance component is a DI component and will not available in bigdata spark job, so far.

Have you tried to utilize a DI Job to orchestrate your spark job by using tRunJob component?

Best regards




Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.


Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.