logic for date on batch processing

Highlighted
Six Stars

logic for date on batch processing

hi all,

 

i need logic to develop from source data that job should read starting date and till ending date and this should be processed 

example 

we are running batch job on 12:00 till next date 

15/10/2018 12:00  till 16/10/2018 12:00

i need the logic how can we achieve

 

thanks

 


Accepted Solutions
Highlighted
Employee

Re: logic for date on batch processing

Hi,

 

    My understanding is that you are trying to run a delta data extraction process. If you are using a Database, the advise will be to store the last run time in the DB and pick the last run time from DB as the first first step. You can store this data into a context variable and continue the processing. In this way, you will not miss any data if you are manually running beyond the normal schedule time.

 

    Now, if you want to capture the 12 am time of current day, you can create a context variable and populate the current day to it (using Talend or java date functions). 

 

If the answer has helped you, could you please mark the topic as resolved? Kudos are also welcome :-)

 

Warm Regards,

 

Nikhil Thampi

View solution in original post


All Replies
Highlighted
Employee

Re: logic for date on batch processing

Hi,

 

    My understanding is that you are trying to run a delta data extraction process. If you are using a Database, the advise will be to store the last run time in the DB and pick the last run time from DB as the first first step. You can store this data into a context variable and continue the processing. In this way, you will not miss any data if you are manually running beyond the normal schedule time.

 

    Now, if you want to capture the 12 am time of current day, you can create a context variable and populate the current day to it (using Talend or java date functions). 

 

If the answer has helped you, could you please mark the topic as resolved? Kudos are also welcome :-)

 

Warm Regards,

 

Nikhil Thampi

View solution in original post

Highlighted
Six Stars

Re: logic for date on batch processing

nikhil

 

could you help me how to do this 

 

thanks

Highlighted
Employee

Re: logic for date on batch processing

Hi Mahesh,

 

       I would say, try to create a sample job based on the above details and if you are stuck, please post the job screen shot and other details in the forum. In this way, you will learn the finer aspects of Talend tool yourself :-)

 

        Also talk to your architects and designers about how they want to fetch the delta data. Give all the options discussed above with them so that they can take an informed decision on job creation.

 

Warm Regards,

 

Nikhil Thampi

 

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Talend Cloud Developer Series – Defining Metadata

This video focuses on different methods of adding metadata to a job in Talend Cloud

Watch Now

Talend Cloud Developer Series – Updating Context Variables

This video will show you how to add context parameters to a job in Talend Cloud

Watch Now

Talend Cloud Developer Series – Deploying First Job to Cloud

This video will show you how to run a job in Studio and then publish that job to Talend Cloud

Watch Now