One Star

Where clause on Talend

I extract data from a big DataWarehouse Oracle.
Currently, I work on a table which contains about 10 000 000 rows. So I need to filter this data. In Sql I use the where clause but, with Talend I want use tFilterRow. I see a big problem :
when I transmit my filters in SQL with the where clause it's the server which stocks Oracle which manages the filters. But when I use tFilterRow, in fact all the rows are extract and they are filtered in "local". The exection is very different (30 seconds versus 3 hours...)
It's necessary to give my filter in SQL at Oracle or I need to param something ?
6 REPLIES
One Star

Re: Where clause on Talend

Hello,
Said ad-hoc (I´m a newbie to TOS):
1.- Customizing application env. (like increasing JVM´s Heap for such a volume of data)
2.- For TOS developer: ?Buffering data within Java routines, which handle high volume data, whereby code sets the size of buffer in dependency of configured heap size, set by user ?.
Isn´t a good practice to move the TOS toward easy processing of Higher Requirements?
aski
Employee

Re: Where clause on Talend

Hello,
I would really do the where in the SQL query.
Regards,
One Star

Re: Where clause on Talend

Hi mhirt,
Sorry but I don't understood your answer.
You want say that improve SQL builder (particulary where clause) for the next version of Talend is in prevision or you want say that I need to develop the component ?
Thx for your answer.
Employee

Re: Where clause on Talend

If you ask tOracleInput to read 10 000 000 rows, in all the cases, it will be very long.
Simply add your filter to your sql request in tOracleInput.
Why are you speaking about "Improving SQLBuilder" ?
One Star

Re: Where clause on Talend

For me,
an ETL like Talend is very interisting because the user need to param your soft by a GUI. But in SQL Builder, for the where clause you need to add manually the where condition. For me, I think it will be very interisting if SQL Builder purpose the tFilterRow component in order to create the SQL request with where clause.
Do you see what I want say ?
Employee

Re: Where clause on Talend

You mean adding filters in SQBuilder configurable like the filters of SQLBuilder ?
It can be an interesting feature.
Do you mind creating a feature request on our bugtracker ?
Regards,