Main job with jobs in sequence keeps running

Highlighted
Five Stars

Main job with jobs in sequence keeps running

Hi

I have two jobs and each of these jobs are getting messages from Kafka topic and pushing into DB. These jobs terminate when there are no messages on kafka topic after 10 secs. Individually both these jobs works as expected. Now I have created a main job that simply calls these jobs in sequence tJob1 - tJob2. When I run this main job I expect the job to terminate automatically after 20 secs if no messages on topic. But this main job keeps running for 6 mins before terminating. Any pointers?

 

Starting job I900_901_Plan_Main at 11:35 11/12/2019.


[statistics] connecting to socket on port 3815
[statistics] connected
tRunJob_1 in I900_901_Plan_Main call mars_project.test_consume_job_0_1.test_consume_job with:

[--father_pid=enEB33, --root_pid=enEB33, --father_node=tRunJob_1, --context=Default, --stat_port=3815, --parent_part_launcher=JOB:I900_901_Plan_Main/NODE:tRunJob_1]

tRunJob_2 in I900_901_Plan_Main call mars_project.test_consume_job2_0_1.test_consume_job2 with:

[--father_pid=enEB33, --root_pid=enEB33, --father_node=tRunJob_2, --context=Default, --stat_port=3815, --parent_part_launcher=JOB:I900_901_Plan_Main/NODE:tRunJob_2]

312957 milliseconds
[statistics] disconnected

Job I900_901_Plan_Main ended at 11:41 11/12/2019. [exit code=0]

Highlighted
Six Stars

Re: Main job with jobs in sequence keeps running

The condition you put in job1 and job2 to check messages from kafka topic, put that in the main job instead. See if this approach works.

Any screenshots of the job would be more helpful.

Cheers
Highlighted
Five Stars

Re: Main job with jobs in sequence keeps running

How can I include topic specific setting in main job. See screenshot below

Main Job:

 

Main_Job.PNG

Child Job1 (Job2 is exact clone of job1 but listens to different topic):

Job1.PNG

Highlighted
Five Stars

Re: Main job with jobs in sequence keeps running

In fact what I noticed is that Sub Job is not at all executed in Main Job. Its same behavior when I have the Subjob components entirely in  the same job. Is it a bug that tKafkaInput does not work when used in conjunction?

See below second subjob just gets stuck waiting to be executed

Subjob_issue.PNG

Highlighted
Employee

Re: Main job with jobs in sequence keeps running

Hi,

 

   I could see that the first subjob got executed successfully.

 

   Could you please check whether the job is running fine when we are trying to print it in Console instead of trying to write to DB? Please also check whether Die on Error option is selected in tDBOutput component.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Highlighted
Five Stars

Re: Main job with jobs in sequence keeps running

Yes tDbOutput die on error is checked.

I have tried with just tKafkaInput and logging the output to screen but its same result (i.e. gets stuck in second subjob).

If I remove tKafkaInput keeping the tDBOutput component along with tJava it works fine.

This indicates that somehow use of tKafkaInput subsequently causes program to wait even when there are messages to be picked up in second topic.

Highlighted
Five Stars

Re: Main job with jobs in sequence keeps running

I saw similar topic but without solution

https://www.talendforge.org/forum/viewtopic.php?id=45987

 

Any clue if it could be a bug in Talend?

Highlighted
Five Stars

Re: Main job with jobs in sequence keeps running

Any help in this regard is appreciated.

 

Thanks.

Highlighted
Employee

Re: Main job with jobs in sequence keeps running

Hi,

 

   I could see that the first Kafka input seems to be working for you. Could you please double check whether there is any difference between connection details? Could you please also check whether the subjobs are working fine when you are trying to run them as separate Talend jobs?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Highlighted
Five Stars

Re: Main job with jobs in sequence keeps running

Yes its same connection in both subjobs and as I provided my comment earlier

These jobs terminate when there are no messages on kafka topic after 10 secs. Individually both these jobs works as expected.

Highlighted
Employee

Re: Main job with jobs in sequence keeps running

Hi,

 

    I am not pretty sure about this behavior. Lets check with @xdshi 

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now