I created one Talend job for getting data from a SQL server database which is hosted in a server, then process data and push in to MySQL database in Amazon RDS. Previously we hosted the Job at EC2 instance in a scheduled manner. It was working fine. Now we have a change in architecture and was forced to make the job scheduled in our network, pull data from database and push it to Amazon. But now the speed of the ETL is reduce to 1 row/sec which is not practical. We have to process more than 5 lakh rows a day. The components we have used is sql server input component and mysql output component. We are using 'Insert or Update' in output component since we have to insert and update the rows. it would be so helpful if any one cam help me out.
By the term change in architecture, i meant that change in the environment which the job was scheduled. Previously it was running in Amazon EC2 instance and uploading data in the database hosted in Amazon RDS instance. Now the job is scheduled in our internal network and uploads data in to the RDS database. Previously the speed of the job was like it used to get completed by 4 hours. Now the job still continues to run even after 2 days. Below given the screenshot of a small portion of entire job. All that we used in the entire job is the repetition of sub-jobs as seen in the screenshot. We are writing different queries to fetch data and uploads it to various tables. It would be great if you can provide any valuable feedbacks.
Hi, How did you set Commit Every in t<DB>output component? The job hangs without error?(timeout...) What happens when you blackhole your output (i.e. sending output to an empty tJavaRow rather than a MySQLOutput)? Best regards Sabrina
-- Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
It is not throwing any exception. If we are connecting this to our local database speed is better. But i would like to know is there any better way to improve the performance from 1 row/s to at least 500 rows/sec