Validate thousand columns of a data row

Highlighted
Five Stars

Validate thousand columns of a data row

Hi everyone,
I'm new to Talend and I need some supports.
The requirement here:
Get data from Redshift, export valid/error data to CSV file.
Check all columns of each data rows if it match with schema of an existing table.
Export error data to a CSV file, infos need to export are the value of primary key, column name and error messages.
Export the valid data row to other one.
Conditions are:
- Check data type
- Check data lenght, if it's a float check the lenght after the comma
- Check date pattern
- Check if the primary key is duplicated with any one in the getted data list.

* If a column has more error, only export to one row in CSV, error mesages are separated by comma.
* If one data row has more error columns, export to multiple rows.

** Here a big note: a data row has more than 1000 column.

Can you give some solution for this requirement ?
Sorry for my English Smiley Very Happy Smiley Very Happy
Thank you so much!


Accepted Solutions
Employee

Re: Validate thousand columns of a data row

Use tSchemaComplianceCheck component.  And some additional checks.


All Replies
Employee

Re: Validate thousand columns of a data row

Use tSchemaComplianceCheck component.  And some additional checks.

Five Stars

Re: Validate thousand columns of a data row

Thank you. But my account don't want to use a component to do it, they need us do it by routine, any more sugest ?

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch