Validate thousand columns of a data row

Five Stars

Validate thousand columns of a data row

Hi everyone,
I'm new to Talend and I need some supports.
The requirement here:
Get data from Redshift, export valid/error data to CSV file.
Check all columns of each data rows if it match with schema of an existing table.
Export error data to a CSV file, infos need to export are the value of primary key, column name and error messages.
Export the valid data row to other one.
Conditions are:
- Check data type
- Check data lenght, if it's a float check the lenght after the comma
- Check date pattern
- Check if the primary key is duplicated with any one in the getted data list.

* If a column has more error, only export to one row in CSV, error mesages are separated by comma.
* If one data row has more error columns, export to multiple rows.

** Here a big note: a data row has more than 1000 column.

Can you give some solution for this requirement ?
Sorry for my English Smiley Very Happy Smiley Very Happy
Thank you so much!


Accepted Solutions
Employee

Re: Validate thousand columns of a data row

Use tSchemaComplianceCheck component.  And some additional checks.


All Replies
Employee

Re: Validate thousand columns of a data row

Use tSchemaComplianceCheck component.  And some additional checks.

Five Stars

Re: Validate thousand columns of a data row

Thank you. But my account don't want to use a component to do it, they need us do it by routine, any more sugest ?

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch