Six Stars

Validate thousand columns of a data row

Hi everyone,
I'm new to Talend and I need some supports.
The requirement here:
Get data from Redshift, export valid/error data to CSV file.
Check all columns of each data rows if it match with schema of an existing table.
Export error data to a CSV file, infos need to export are the value of primary key, column name and error messages.
Export the valid data row to other one.
Conditions are:
- Check data type
- Check data lenght, if it's a float check the lenght after the comma
- Check date pattern
- Check if the primary key is duplicated with any one in the getted data list.

* If a column has more error, only export to one row in CSV, error mesages are separated by comma.
* If one data row has more error columns, export to multiple rows.

** Here a big note: a data row has more than 1000 column.

Can you give some solution for this requirement ?
Sorry for my English Smiley Very Happy Smiley Very Happy
Thank you so much!

  • Data Integration
1 ACCEPTED SOLUTION

Accepted Solutions
Employee

Re: Validate thousand columns of a data row

Use tSchemaComplianceCheck component.  And some additional checks.

2 REPLIES
Employee

Re: Validate thousand columns of a data row

Use tSchemaComplianceCheck component.  And some additional checks.

Six Stars

Re: Validate thousand columns of a data row

Thank you. But my account don't want to use a component to do it, they need us do it by routine, any more sugest ?