Job Performance tuning help required incase of genarting extract file from sybase table

Six Stars

Job Performance tuning help required incase of genarting extract file from sybase table

Hi,

 

I am simply generating extract file from sybase table using tSybaseInput and tfileoutputdelimited using tmap and i am not doing any transformation. Table have data around 217 million. can someone help  me to provide performance technique which i should use to get better performance. I am using Talend 6.5.1.

 

Regards,

Rahul

Nine Stars

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Hi @rahuljan ,

You can use parallelization and increase the number the of child process(threads) , also what is the exact time the job is running.???

Pls do get to Community, if necessary.

Thanks,

Ankit.

Employee

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Hi Rahul,

 

     One way of increasing the throughput of writing data is to use tparallelize and fetch the data in parallel for diferent data sets from source database. For example, you can run two or three subjobs from tparallelize component which fetches data for different date range and writing to target file. Please remember to add the append mode on so that multiple proccess can write at same time.

 

     You will have to increase the memory parameters also for the job to make sure that data is passing without any hiccups.

 

    I would even try to run multiple jobs also to make things faster by using all available firepower in Jobserver :-)

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

Six Stars

Re: Job Performance tuning help required incase of genarting extract file from sybase table

@nikhilthampi  Is there any other solution since using parallel component data will be not in same sequence and that i important in my job?

Six Stars

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Check the below url -

Custom the flush buffer size

 

This is usually there in the Advanced Settings Tab of tFileOutputDelimited

Highlighted
Employee

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Hi,

 

    I agree with @dipanjan93  about buffer size flusing as it is also going to change your data throughput.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Hi @rahuljan ,

 

How about the query performance if we use limit clause. If it is good, then we can export data without using tParallel component.

 

Ex Query: 

select * from tablename limit 1000000;
Employee

Re: Job Performance tuning help required incase of genarting extract file from sybase table

@hpamidipola 

 

Using Limits might not be a good idea in this case as we have to export the data from source table even if the records are more than the limit level. If we specify a hardcoded limit, the data after the limit will be ignored and will eventually result in data loss or mismatch.

 

@rahuljan 

 

  If the details provided by the members have helped you, could you please mark the topic as resolved? It will help other community members also during their reference.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads