discard rows with a number of null columns

Four Stars

discard rows with a number of null columns

Hi all,


I'm new to Talend. 


Here is my requirement: 

we need to discard rows of a table when the 25% of their columns is null.




            col1 | col2 | col3 | col4


row1    null  | 123  | null  | null


row2    456  | 123  | 789  | 000



row1 will be discarded 

row2 is good and pass the validation


The steps would be:

1) calculate the number of columns of a table

2) for every row, to check which column is null and count the number of null columns

3) if that number exceed the limit, then discard the row


Any suggestion on how to achieve it?

So far, I know how to get the first step, executing a SQL query.


For the second step I know the tMap component can be used to check rows, but then how to compare the numbers of columns dinamically? Maybe using a tJava component and writing code?




Tags (1)