Four Stars

Restart ability in Talend with check points parallely but not sequentially

Error.JPG

 

only requirement i have, in example shown above  if i understood correctly if any thing failed at first check point it self it won't continue  subsequent sub jobs since we used the condition ONSUBJOB OK.  My question is why we want to block subsequent sub jobs even it might not have any issues/error's?  --- Which means this process is running Sequentially. My requirement is even if first job failed at first check point it should not impact other sub jobs,  other sub jobs has to run as is . When we restart again it has to pick up only failed job, it might be first  job or Last Job.  so  we need to make it process as  parallel execution not sequentially. Hope i am clear in my requirement still if you have any concerns please let me know i will try to put this is different way .

 

I tried with some draft design like below 

 

t java (reading parameters from property file) ----- tparalleize(paralleize)-----tcheckstart1---tcheckpoint1---trunjob1---tcheckend 1

from same  tparalleize in above design one more link like

-----tcheckstart2---tcheckpoint2---trunjob2---tcheckend 2

i am getting below error's

Exception in component tCheckpointEnd_2
java.lang.RuntimeException: tCheckpointEnd_2: you must have a tCheckpointStart and at least one tCheckpoint before ending a checkpoint series.

Exception in component tCheckpointStart_2
java.lang.RuntimeException: tCheckpointStart_2: you must end the current checkpoint series before starting a new one.

can you please help with above error

 

can you please let me know  parallel approach with check points  is possible or not? or i need to follow any different approach for restart ability in talend .. If possible can you help with sample design 

 

Thanks in Advance

  • Data Integration
4 REPLIES
Four Stars

Re: Restart ability in Talend with check points parallely but not sequentially

Thanks for your quick response. I will add you to my skype. Hope i will hear a proposed solution as soon as possible

Seven Stars

Re: Restart ability in Talend with check points parallely but not sequentially

sai8166 wrote:

My question is why we want to block subsequent sub jobs even it might not have any issues/error's?

because - this is logic of this Job

It user created component, which really check saved statuses one-by-one

 

You can achieve same result by different ways, but main idea always will be same - checkpoint it is saved sate of Chain Step.

If You have parallel steps - do not use chain, use semaphore for each of them independent

-----------
Four Stars

Re: Restart ability in Talend with check points parallely but not sequentially

Thanks for your Reply.

 

I want it to be parallel executed. If First check point it self failed i need to continue subsequent Sub Jobs. When next time Triggers it has to pick  up only failed one's . Is there  a way? can you please send a draft design for reference if  there is way to make it as parallel executed

Seven Stars

Re: Restart ability in Talend with check points parallely but not sequentially

 

Screen Shot 2017-05-10 at 12.43.24 PM.png

 

 

the logic, could be any - this simple example mean

successful Job create lock file and delete at the end, any time when it start - first check "is lock file there?" if yes - it mean last iteration was finished with error.

You can extend logic like - is lock file there and timestamp inside file equal today? and etc

-----------