I extract data from a big DataWarehouse Oracle. Currently, I work on a table which contains about 10 000 000 rows. So I need to filter this data. In Sql I use the where clause but, with Talend I want use tFilterRow. I see a big problem : when I transmit my filters in SQL with the where clause it's the server which stocks Oracle which manages the filters. But when I use tFilterRow, in fact all the rows are extract and they are filtered in "local". The exection is very different (30 seconds versus 3 hours...) It's necessary to give my filter in SQL at Oracle or I need to param something ?
Hello, Said ad-hoc (I´m a newbie to TOS): 1.- Customizing application env. (like increasing JVM´s Heap for such a volume of data) 2.- For TOS developer: ?Buffering data within Java routines, which handle high volume data, whereby code sets the size of buffer in dependency of configured heap size, set by user ?. Isn´t a good practice to move the TOS toward easy processing of Higher Requirements? aski
Hi mhirt, Sorry but I don't understood your answer. You want say that improve SQL builder (particulary where clause) for the next version of Talend is in prevision or you want say that I need to develop the component ? Thx for your answer.
For me, an ETL like Talend is very interisting because the user need to param your soft by a GUI. But in SQL Builder, for the where clause you need to add manually the where condition. For me, I think it will be very interisting if SQL Builder purpose the tFilterRow component in order to create the SQL request with where clause. Do you see what I want say ?