Four Stars

Data Insertion

Hi Everyone , 

i am new to talend. While understanding Data Insertion in Talend , come across a doubt.

 While inserting Data by using toracleoutput or any other bulk components,  due to any reason if Connection failure is happen how the data will be committed and how system knows to process remaining rows in the next re-run. 

1 ACCEPTED SOLUTION

Accepted Solutions
Six Stars

Re: Data Insertion

On using tOracleOutput, in Advanced Settings:

"Commit every": define number of rows to be completed before committing batches of rows together into database.
"Batch Size": Specify the number of records to be processed in each batch.

If connection failed/job failed, then on re-running the job, it will re-insert the committed records to oracle table.
If need to prevent this, need to put tOraclerollback in after the subjob runs.

tOracleConnection -- tOracleInput -> tmap -> toracleOutput / --OnsubjobOk --> tOracleCommit
                                                                                               \ --OnsubjobError --> tOracleRollback

If jobs fails it will rollback all the transaction.

 


Regards,
Mohit
3 REPLIES
Fifteen Stars

Re: Data Insertion

It is your job as a data integration specialist to build jobs that will support that. The tools in Talend are there, but there is no magic button to do it for you. As an example, you could keep a log table to log success/failure, number of rows, watermarks (high/low) of your source data, etc. When your job starts you look at this table to get history information, then write to this table to acknowledge a start. Then when your job finishes successfully or with a failure, you store information in that table to help you with your next run.

Rilhia Solutions
Six Stars

Re: Data Insertion

On using tOracleOutput, in Advanced Settings:

"Commit every": define number of rows to be completed before committing batches of rows together into database.
"Batch Size": Specify the number of records to be processed in each batch.

If connection failed/job failed, then on re-running the job, it will re-insert the committed records to oracle table.
If need to prevent this, need to put tOraclerollback in after the subjob runs.

tOracleConnection -- tOracleInput -> tmap -> toracleOutput / --OnsubjobOk --> tOracleCommit
                                                                                               \ --OnsubjobError --> tOracleRollback

If jobs fails it will rollback all the transaction.

 


Regards,
Mohit
Six Stars

Re: Data Insertion

@rhall_2_0, I agree with your point.


Regards,
Mohit