Write huge text files tfileoutputdelimited

Highlighted
Five Stars

Write huge text files tfileoutputdelimited

Hello,

I have a need to run a query on Oracle that results more than 500k records and write to a text file. I have just used two components as below

toracleinput -> tfileoutputdelimited

The current write speed is ~250 rows/second. How can I increase the speed of data write? I have enterprise subscription and "enable parallel execution" on tfileoutputdelimited is greyed out as it seems to be available only for map/reduce jobs.

Please suggest.

 

Thanks,

Surya


Accepted Solutions
Six Stars

Re: Write huge text files tfileoutputdelimited

Hello,

 

Firstly, maybe you can tick the "use a cursor" function in the advanced setting in you tOracleInput component. And try different numbers.


All Replies
Six Stars

Re: Write huge text files tfileoutputdelimited

Hello,

Firstly, maybe you can tick the "use a cursor" function in the advanced setting in you tOracleInput component. And try different number.

Six Stars

Re: Write huge text files tfileoutputdelimited

Hello,

 

Firstly, maybe you can tick the "use a cursor" function in the advanced setting in you tOracleInput component. And try different numbers.

Fifteen Stars TRF
Fifteen Stars

Re: Write huge text files tfileoutputdelimited

Hi,

What if you don't push the result to a file or if you execute the same query from SQL (out of TOS)?


TRF
Community Manager

Re: Write huge text files tfileoutputdelimited

The problem is with your query or your database component config. I've just tested writing a million rows of 600 chars to a flat file and got a performance of 49465 rows per second written.....on a reasonably middle of the road machine. Remove your fileoutput component and add a tJavaFlex (with nothing configured.....just to see the rows per second metrics). How long does that take? Now tweak your query component and your query.

Five Stars

Re: Write huge text files tfileoutputdelimited

I was just doing a select * from Table.

Five Stars

Re: Write huge text files tfileoutputdelimited

SQL runs fine. It's just with the export speed

Community Manager

Re: Write huge text files tfileoutputdelimited

The SQL might run fine, but the Talend database component may not be configured adequately. As I have said, you can write thousands of rows per second to a flat file. That will not be the bottleneck. Have you tested removing the fileoutput component and just connecting to a tJavaFlex to compare the performance?

 

Can you show us screenshots of your job and your db component config (including the Advanced tab)?

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch