Data Insertion

Highlighted
Four Stars

Data Insertion

Hi Everyone , 

i am new to talend. While understanding Data Insertion in Talend , come across a doubt.

 While inserting Data by using toracleoutput or any other bulk components,  due to any reason if Connection failure is happen how the data will be committed and how system knows to process remaining rows in the next re-run. 


Accepted Solutions
Highlighted
Seven Stars

Re: Data Insertion

On using tOracleOutput, in Advanced Settings:

"Commit every": define number of rows to be completed before committing batches of rows together into database.
"Batch Size": Specify the number of records to be processed in each batch.

If connection failed/job failed, then on re-running the job, it will re-insert the committed records to oracle table.
If need to prevent this, need to put tOraclerollback in after the subjob runs.

tOracleConnection -- tOracleInput -> tmap -> toracleOutput / --OnsubjobOk --> tOracleCommit
                                                                                               \ --OnsubjobError --> tOracleRollback

If jobs fails it will rollback all the transaction.

 


Regards,
Mohit

View solution in original post


All Replies
Highlighted
Community Manager

Re: Data Insertion

It is your job as a data integration specialist to build jobs that will support that. The tools in Talend are there, but there is no magic button to do it for you. As an example, you could keep a log table to log success/failure, number of rows, watermarks (high/low) of your source data, etc. When your job starts you look at this table to get history information, then write to this table to acknowledge a start. Then when your job finishes successfully or with a failure, you store information in that table to help you with your next run.

Highlighted
Seven Stars

Re: Data Insertion

On using tOracleOutput, in Advanced Settings:

"Commit every": define number of rows to be completed before committing batches of rows together into database.
"Batch Size": Specify the number of records to be processed in each batch.

If connection failed/job failed, then on re-running the job, it will re-insert the committed records to oracle table.
If need to prevent this, need to put tOraclerollback in after the subjob runs.

tOracleConnection -- tOracleInput -> tmap -> toracleOutput / --OnsubjobOk --> tOracleCommit
                                                                                               \ --OnsubjobError --> tOracleRollback

If jobs fails it will rollback all the transaction.

 


Regards,
Mohit

View solution in original post

Highlighted
Seven Stars

Re: Data Insertion

@rhall_2_0, I agree with your point.


Regards,
Mohit

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog