Best component or practice to apply data quality rules

Seven Stars

Best component or practice to apply data quality rules

Hi All,

I need to apply certain data quality rules on columns in a row.

However i would like to apply these rules at once and not one after another.

also i need to collect all failures for all the columns.

Can someone pls guide for best practice to do so or best component to use

e.g Table 1 

Columns  ---->A | B | C

DQ Rules ---->Not null | Should be > 0 | Value in list [1,2,3]

now to check not null - we can use tschemaComplianceCheck

should be > 0 - we can use tFilter

Value in list[1,2,3] - we can use tMap

and moreover, these rules has to dynamically applied, for e.g these rules can be stored in oracle table so that we can change the rule whenever required and need not change anything in the job.

Pls help.

 

Thanks

 

Six Stars

Re: Best component or practice to apply data quality rules

Hi vidya,

 

I maybe don't get all your requirements but i think you can do everything in a tMap.

In the expression filter you can put something like :

row.A!=null&&row.B>0&&(row.C=1||row.C=2||row.C=3)

For sure there is a better way to test the column C but this solution he's working i think.

 

Warm regards

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog