I am working on TOS. In this i have created a certain number of jobs and done schedule through 'windows task scheduler'.
My problem is, sometimes the job is getting completed within 2 mins and sometimes the same job is taking more time (approx 2 hrs) to complete the execution.
What is the reason for this and how could it be rectified?
@Rajkumar ,Need to analyze that ,when it took long time how much data was processing comparatively when it took 2 mins to complete and when it was executing longtime other tasks are running.
@Rajkumar ,Ok..how was the network speed when it took long time comparatively to 2 mins time?
and as mentioned when it took long time any other jobs are executing compare to 2 mins run?
@Rajkumar ,when it was taking long time,is it takes time to fetch data?
@Rajkumar ,can you analyse source statistics when that job was running long time and less time?
Q1) How many components you are having inside the talend job.
Q2) How many sub jobs are inside your main job
Q3) what are the sources, targets and lookups used and their types
Use TJava to print system time after each and every component. And share the log of execution to analyze further.
make sure the tables you use are not being locked by other objects. if other user or other job has locked your table. writing to that table will be kept on hold till the lock is released.
Also see how much throughput you are getting for the mapping.
Use bullk load component to load to target
--Do the tables got locked while fetching the data through talend?
--Bulk load is better option, but i am facing some error. 'Access denied'
"Exception in component tDBOutputBulkExec_1_tMBE (New_component)
java.sql.SQLException: Cannot bulk load because the file "C:/Users/*****/Desktop/talend/TOS_DI-20180411_1414-V7.0.1/workspace/mssql_data.txt" could not be opened. Operating system error code 5(Access is denied.).
and also I have one more doubt.
--Do we enable bulk load permission in sql database?
Did you check the DB statistics from your DBA to verify that DB is having enough resources?
I am having doubt about the performance of DB, number of simultaneous connections allowed, usage of indexes etc. (to say a few).
Also please check the memory and CPU utilization of the server where Talend jobs are running (you have already checked the network part).
If its Oracle, I would have asked DBAs to provide AWR and ADDM reports.
I am not sure the equivalent of ADDM report in SQL Server but your DBA can help you in it.
Are you having enough CPU and memory in the machine where you are running Talend jobs? Please refer the recommended Resource size for enterprise products as a bench marks.
You need to have comparable infrastructure in your server like a job server.
@Rajkumar ,to load data using Bulk method ,you should use combination of tOracleOutputBulk and tOracleOutputBulk
@Rajkumar ,if you are using MSSql server then yes.
@Rajkumar ,tmssqloutputbulk will create a Bulk file to load into DB,so may be you may not have access to create BULK file ,please check the access to create a file in that location
@Rajkumar ,yes in DB side you need to do I belive
@Rajkumar ,Yes for the Boolean values it should be true or false. what is your target DB type? based on your DB allows you need to manage that filed as string.
@Rajkumar ,if i understand correctly you need to convert true to 1 and false to 0 .
@Rajkumar ,where you were creating true/false,you can generate 1/0 but the o/p column data type should be string.
@manodwhb, but my need is boolean data type.....so do any other way to reproduce the same in text file also?
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema