Four Stars

scanario

Hi,

 

How to implement below logic.

 

tExcelInput---->tOralceOutput1(table_1)
                           |
                           | (onComponentsOk_1)
                           |
                     tOracleInput ------>tOralceOutput2(table_2)

 

table_1

columns: emono, date_col

               1, 20-07-2017

                2, 20-07-2017

 

table_2:  existing data before load table_1 records from excel file.

emono, date_col,

1, 18-07-2017

 2, 18-07-2017

 

 

Scenario:

1) In between tOralceOutput1 and tOralceOutput2, Need to check below logic.

 

if table_1.date_col <> table_2.date_col

   wait for process to call ( tOracleInput ------>tOralceOutput2(table_2) 

   check every 15min continues(infinity loop)

else

   if table_1.date_col ==  table_2.date_col

       continues to next step ( tOracleInput ------>tOralceOutput2(table_2) 

end if

 

Thanks

Shree

 

  • Data Integration
4 REPLIES
Eleven Stars

Re: scanario

I have concerns about the required logic here. You have, in your pseudo code, a line that says....

check every 15min continues(infinity loop)

Talend jobs are batch. They should start, process and end. It is not a great idea to leave them essentially polling every 15 mins. To do this you may want to use Talend ESB. 

 

The other concern with this is the expectation that your dates will match exactly. Are they dates or Strings representing dates? If it is the latter, then this will work (although don't compare them with "="). Otherwise you may want to truncate your dates to just days, since the whole date will go down to milliseconds.

 

However, before you get to the actual comparisons, you do need to think about whether you have to have a job that *could* run forever. What is the business the requirement you are trying to solve?

Rilhia Solutions
Four Stars

Re: scanario

Hi

 

Thanks for your input.

 

Yes, it is business requirement. Whole job i cant share if want share any gmail ID i can.

 

Coming to point 1.

 

I have concerns about the required logic here. You have, in your pseudo code, a line that says....
check every 15min continues(infinity loop)

Talend jobs are batch. They should start, process and end. It is not a great idea to leave them essentially polling every 15 mins. To do this you may want to use Talend ESB. 

Ans: Let me check with Talend ESB. Any specific component in ESB i need to work around.

 

The other concern with this is the expectation that your dates will match exactly. Are they dates or Strings representing dates? If it is the latter, then this will work (although don't compare them with "="). Otherwise you may want to truncate your dates to just days, since the whole date will go down to milliseconds.

 

Data type is Date and Oracle DB.

 

Please let me know any further input or if you have any question?

 

Thanks

Shree

 

Eleven Stars

Re: scanario

With regard to the ESB you will want to look into Routes. Do a bit of reading on Apache Camel routes. It isn't as straight forward as the ETL tool though. You will be wanting to look at this http://camel.apache.org/sql-component.html

 

Another way you could approach this (which would be safer if using an ETL job) is to schedule the whole job to run every 5 or so minutes. This would remove the need for using the ESB, but would make the job a lot safer. Something to consider. This really does depend on your business requirement I guess. 

 

If you are using Oracle, you can output the Date data truncated to yyyyMMdd which would solve potential matching issues.

Rilhia Solutions
Four Stars

Re: scanario

Thanks for inputs.

 

Let me check with team and will go through Apache Camel routes.

 

Regards

shree