Data Flow control mechanisms in Talend?

One Star

Data Flow control mechanisms in Talend?

Hi everyone,
I have a job that implements a simple web service.
The job should do this:
1. Retrieve some of the request parameters and memorize them
2. Use those parameters in a SQL query (its where-clauses) to fetch data
3. Filter that data
4. Create some response
I have problems with filtering in Talend, though; here's a sample of the process that needs to happen:
TAKE all the data you fetched from the database
SPLIT the result set into 2 lists:
List A contains all result set records that match a certain pattern (that have certain values in a certain field)
List B contains the rest
IF List A is empty, use List B
IF the List (A or B - depending on the previous step) has more than 20 records, throw a fault
SORT that List by the Date in field A or field B (Take field B if field A is empty)
Take the top 1 record
If field C of that top 1 record contains a certain value, throw a fault
Take field D and create a response
That's a simple filter algorithm that requires not only data flow (sorting, simple filtering) but also flow control (IF/THEN-conditions, branches, counting of number of rows etc.)
But how can that be done in Talend?
I find the filter and row sort mechanisms too rudimentary (or I don't understand them fully).
And I have not seen anything that implements conditions in an intuitive way.
Any ideas?
Thanks
Matt
Four Stars

Re: Data Flow control mechanisms in Talend?

Hi Matt,
TAKE all the data you fetched from the database
>> take database Input component
SPLIT the result set into 2 lists:
>> Use tReplicate component
List A contains all result set records that match a certain pattern (that have certain values in a certain field)
>> Use tFilter with appropriate condition for A output connect output to tHash component
List B contains the rest
>> use tFilter with another condition for B output connect output to tHash component
IF List A is empty, use List B
>> Use RunIf clause from tHash component use appropriate condition
IF the List (A or B - depending on the previous step) has more than 20 records, throw a fault
>> Use RunIf clause from tFilter use appropriate condition and connect to tWarn/tDie or generate your exception using tJava
SORT that List by the Date in field A or field B (Take field B if field A is empty)
>> Use RunIf and take appropriate tHash component and get the records
Take the top 1 record
>> Need to use again tFilter to get the Top record based on the data values
If field C of that top 1 record contains a certain value, throw a fault
Take field D and create a response
>> Use context variables during your job for conditions to be used in RunIf ...
Hope you got some idea...
Thanks
vaibhav