only requirement i have, in example shown above if i understood correctly if any thing failed at first check point it self it won't continue subsequent sub jobs since we used the condition ONSUBJOB OK. My question is why we want to block subsequent sub jobs even it might not have any issues/error's? --- Which means this process is running Sequentially. My requirement is even if first job failed at first check point it should not impact other sub jobs, other sub jobs has to run as is . When we restart again it has to pick up only failed job, it might be first job or Last Job. so we need to make it process as parallel execution not sequentially. Hope i am clear in my requirement still if you have any concerns please let me know i will try to put this is different way .
I tried with some draft design like below
t java (reading parameters from property file) ----- tparalleize(paralleize)-----tcheckstart1---tcheckpoint1---trunjob1---tcheckend 1
from same tparalleize in above design one more link like
i am getting below error's
Exception in component tCheckpointEnd_2
java.lang.RuntimeException: tCheckpointEnd_2: you must have a tCheckpointStart and at least one tCheckpoint before ending a checkpoint series.
Exception in component tCheckpointStart_2
java.lang.RuntimeException: tCheckpointStart_2: you must end the current checkpoint series before starting a new one.
can you please help with above error
can you please let me know parallel approach with check points is possible or not? or i need to follow any different approach for restart ability in talend .. If possible can you help with sample design
Thanks in Advance
Thanks for your quick response. I will add you to my skype. Hope i will hear a proposed solution as soon as possible
My question is why we want to block subsequent sub jobs even it might not have any issues/error's?
because - this is logic of this Job
It user created component, which really check saved statuses one-by-one
You can achieve same result by different ways, but main idea always will be same - checkpoint it is saved sate of Chain Step.
If You have parallel steps - do not use chain, use semaphore for each of them independent
Thanks for your Reply.
I want it to be parallel executed. If First check point it self failed i need to continue subsequent Sub Jobs. When next time Triggers it has to pick up only failed one's . Is there a way? can you please send a draft design for reference if there is way to make it as parallel executed
the logic, could be any - this simple example mean
successful Job create lock file and delete at the end, any time when it start - first check "is lock file there?" if yes - it mean last iteration was finished with error.
You can extend logic like - is lock file there and timestamp inside file equal today? and etc
Though i have not posted this question, i searched forum on achieving restart feature in Talend. I took your suggestion but implemented in a different way per my requirement. But the base is same as creating a kind of ".done" and ".notdone" files to achieve the restartability feature. I would like to thank you for this solution and saved my time for not duplicating the question!
Did you find any solution when jobs are running parallely kindly let me know if u have a solution
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Create systems and workflow to manage clean data ingestion and data transformation.