Five Stars

Write huge text files tfileoutputdelimited

Hello,

I have a need to run a query on Oracle that results more than 500k records and write to a text file. I have just used two components as below

toracleinput -> tfileoutputdelimited

The current write speed is ~250 rows/second. How can I increase the speed of data write? I have enterprise subscription and "enable parallel execution" on tfileoutputdelimited is greyed out as it seems to be available only for map/reduce jobs.

Please suggest.

 

Thanks,

Surya

1 ACCEPTED SOLUTION

Accepted Solutions
Six Stars

Re: Write huge text files tfileoutputdelimited

Hello,

 

Firstly, maybe you can tick the "use a cursor" function in the advanced setting in you tOracleInput component. And try different numbers.

7 REPLIES
Six Stars

Re: Write huge text files tfileoutputdelimited

Hello,

Firstly, maybe you can tick the "use a cursor" function in the advanced setting in you tOracleInput component. And try different number.

Six Stars

Re: Write huge text files tfileoutputdelimited

Hello,

 

Firstly, maybe you can tick the "use a cursor" function in the advanced setting in you tOracleInput component. And try different numbers.

Thirteen Stars TRF
Thirteen Stars

Re: Write huge text files tfileoutputdelimited

Hi,

What if you don't push the result to a file or if you execute the same query from SQL (out of TOS)?


TRF
Fifteen Stars

Re: Write huge text files tfileoutputdelimited

The problem is with your query or your database component config. I've just tested writing a million rows of 600 chars to a flat file and got a performance of 49465 rows per second written.....on a reasonably middle of the road machine. Remove your fileoutput component and add a tJavaFlex (with nothing configured.....just to see the rows per second metrics). How long does that take? Now tweak your query component and your query.

Rilhia Solutions
Five Stars

Re: Write huge text files tfileoutputdelimited

I was just doing a select * from Table.

Five Stars

Re: Write huge text files tfileoutputdelimited

SQL runs fine. It's just with the export speed

Fifteen Stars

Re: Write huge text files tfileoutputdelimited

The SQL might run fine, but the Talend database component may not be configured adequately. As I have said, you can write thousands of rows per second to a flat file. That will not be the bottleneck. Have you tested removing the fileoutput component and just connecting to a tJavaFlex to compare the performance?

 

Can you show us screenshots of your job and your db component config (including the Advanced tab)?

Rilhia Solutions