I need to apply certain data quality rules on columns in a row.
However i would like to apply these rules at once and not one after another.
also i need to collect all failures for all the columns.
Can someone pls guide for best practice to do so or best component to use
e.g Table 1
Columns ---->A | B | C
DQ Rules ---->Not null | Should be > 0 | Value in list [1,2,3]
now to check not null - we can use tschemaComplianceCheck
should be > 0 - we can use tFilter
Value in list[1,2,3] - we can use tMap
and moreover, these rules has to dynamically applied, for e.g these rules can be stored in oracle table so that we can change the rule whenever required and need not change anything in the job.
I maybe don't get all your requirements but i think you can do everything in a tMap.
In the expression filter you can put something like :
For sure there is a better way to test the column C but this solution he's working i think.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema