Records missing in tOracleInput

Highlighted
Five Stars

Records missing in tOracleInput

Dear team

   I had designed a talend job (in screenshot "Parent_job"), which has a sub job called DT_P_Oracle (in screenshot "Sub_job"). In the sub job I had used 4 tOracleInput where I had divided the month in where clause. Some records randomly missing in table (tOracleInput) read itself (once in a week) which is random in any one of the four sub jobs in screenshot "Sub_job". Please provide solution for this

 

note: all jobs were running parallelly


Accepted Solutions
Twelve Stars

Re: Records missing in tOracleInput

can you check your queries?
is your db on production (with update insert on it)?

Francois Denis

Tag as "solved" for others! Kudos to thanks!


All Replies
Twelve Stars

Re: Records missing in tOracleInput

We do not have enough information to help you.
Can you give us the SQL queries with where clause.
tOracleInput read itself ?what do you mean?
What is your commit method?
I did not see any parallel configuration into your job?

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: Records missing in tOracleInput

Thanks for your reply,

I had attached tOracleInput query in "select Query.txt" file, where I had changed "dt.posted_date" values in tOracleInput_1, tOracleInput_2, tOracleInput_3, tOracleInput_4.

 

"tOracleInput read itself ? what do you mean?" -> I am trying to say that the records (nearly 80 Lakhs records of the total - 5 crores) were missed during the reading process in Oracle DB (tOrackeInput) sometimes. But most of the time, it works well.

 

"What is your commit method?" -> I had attached the screenshot "commit method".

 

"I did not see any parallel configuration into your job?" -> I had enabled multi-thread execution in the main job, and four sessions were opened at back-end at a time for reading data.

 

Twelve Stars

Re: Records missing in tOracleInput

when SBITABLETRUNK had run DT_P_Oracle is runing
I think SBITABLETRUNK trunk table.
DT_P_Oracle is configured to run 4 threads. Is there a configuration to split the job? or are they doing the same job?
I think your problem is here!

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: Records missing in tOracleInput

There's no configuration to split the job except multi-thread execution, All the four threads were doing the same job with a difference in where clause's posted_date range (refer "select Query.txt" in previous attachment).

 

will there be any problem in multi-thread execution?

Twelve Stars

Re: Records missing in tOracleInput

I prefer to use multi thread on tRunJob not inside job.
https://help.talend.com/reader/mhqCkTBnin7IXmJBUJoocQ/6iQsGrBmZWe~mxK9aTY7RA
in Your case you are runing 4 multiple join query's
to read data of the same table.
you cannot garanti unicity and fullfill of datas.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: Records missing in tOracleInput

Hi Denis

 

I had split the four sub-jobs and used tParallelize with four tRunJob. still, 47 Lakhs records missing in the 2nd job (this time), do you have any other suggestions

Twelve Stars

Re: Records missing in tOracleInput

can you check your queries?
is your db on production (with update insert on it)?

Francois Denis

Tag as "solved" for others! Kudos to thanks!

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog