Job Performance tuning help required incase of genarting extract file from sybase table

Five Stars

Job Performance tuning help required incase of genarting extract file from sybase table

Hi,

 

I am simply generating extract file from sybase table using tSybaseInput and tfileoutputdelimited using tmap and i am not doing any transformation. Table have data around 217 million. can someone help  me to provide performance technique which i should use to get better performance. I am using Talend 6.5.1.

 

Regards,

Rahul

Eight Stars

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Hi @rahuljan ,

You can use parallelization and increase the number the of child process(threads) , also what is the exact time the job is running.???

Pls do get to Community, if necessary.

Thanks,

Ankit.

Employee

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Hi Rahul,

 

     One way of increasing the throughput of writing data is to use tparallelize and fetch the data in parallel for diferent data sets from source database. For example, you can run two or three subjobs from tparallelize component which fetches data for different date range and writing to target file. Please remember to add the append mode on so that multiple proccess can write at same time.

 

     You will have to increase the memory parameters also for the job to make sure that data is passing without any hiccups.

 

    I would even try to run multiple jobs also to make things faster by using all available firepower in Jobserver :-)

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)
Five Stars

Re: Job Performance tuning help required incase of genarting extract file from sybase table

@nikhilthampi  Is there any other solution since using parallel component data will be not in same sequence and that i important in my job?

Six Stars

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Check the below url -

Custom the flush buffer size

 

This is usually there in the Advanced Settings Tab of tFileOutputDelimited

Employee

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Hi,

 

    I agree with @dipanjan93  about buffer size flusing as it is also going to change your data throughput.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)
Four Stars

Re: Job Performance tuning help required incase of genarting extract file from sybase table

Hi @rahuljan ,

 

How about the query performance if we use limit clause. If it is good, then we can export data without using tParallel component.

 

Ex Query: 

select * from tablename limit 1000000;
Highlighted
Employee

Re: Job Performance tuning help required incase of genarting extract file from sybase table

@hpamidipola 

 

Using Limits might not be a good idea in this case as we have to export the data from source table even if the records are more than the limit level. If we specify a hardcoded limit, the data after the limit will be ignored and will eventually result in data loss or mismatch.

 

@rahuljan 

 

  If the details provided by the members have helped you, could you please mark the topic as resolved? It will help other community members also during their reference.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.