Batch Processing/Batch mode in talend

Nine Stars

Batch Processing/Batch mode in talend

Hi,

How does Talend handle Batch processing?

Does it generate seperate shell scripts with .bat extensions ?? and where do these .bat scripts get stored and executed??

How is it different from Linux processing?

In almost all Db components, we have enable parallel execution --> if enabled, how does it work??? does it split the records/data into child threads.??

 

Thanks,

Ankit.


Accepted Solutions
Employee

Re: Batch Processing/Batch mode in talend

Hi Ankit,

 

    Please refer the below link for the difference between multi threading and parallellization.

 

https://community.talend.com/t5/Design-and-Development/what-are-the-differences-between-Multi-thread...

 

     Regarding your next query, the parallel option do not restrict the job flow to any order. 

 

      For your last query regarding context variables, below articles will help you.

 

https://community.talend.com/t5/Design-and-Development/Passing-a-value-from-a-parent-Job-to-a-child-...

 

https://community.talend.com/t5/Design-and-Development/Returning-a-value-from-a-child-Job-to-the-par...

 

We are deviating a lot from original query. So I believe we should close the current topic and create another post for queries related to other topics.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

 


All Replies
Employee

Re: Batch Processing/Batch mode in talend

Hi Ankit,

 

      There are lot of queries with different subjects. So lets take one by one.

 

Batch processing is the mode by which you are planning to process data in batch manner like files or DB delta data etc. The other mode is real time processing where we normally employs techniques like web services where response will be instant (or near instant).

Now, you can process batch processing jobs in multiple methods. Running Talend jobs through .sh/.bat files is just one way of processing those files. You can also run these files through TAC if you are using a subscription product. The new version of Talend even allows to create docker files. So it is a method of execution of these jobs. You can refer the details about the various methods from below link.

 

https://help.talend.com/reader/GQMv9RekYnlXzw3tanRDdA/jjIXcdA8eKKSN4YJe17FSg

 

Coming to third query, when you say about Linux processing, I guess you are talking about running the jobs through shell scripts. So it is a sub category of above method where Talend jobs are embedded in Linux shell scripts and you can schedule them through crontab. In Linux itself, if you have setup your TAC installation over a Linux machine, you can run your Talend jobs from there too. 

 

For your query related to DB components, Sabrina has already answered it in another post. Please refer the details below.

 

https://community.talend.com/t5/Design-and-Development/Parallel-Execution/td-p/33850

 

Hope you got answers for all your queries :-)

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

 

Nine Stars

Re: Batch Processing/Batch mode in talend

Hi @nikhilthampi ,

Can you pls clear one final query of mine...

How is Multi-threading execution different from the Regular Parallel Execution??Does it depend on Job design ??

Another query is for example - i have five child jobs(sub jobs), i havent used tparallelize component..but how exactly the execution order commence...does it depend on least number of data which is to moved or complexity of job design etc....???

Another query is for example - i have 1 master comprising of 5 subjobs... now if i want pass a parameter from parent job to child can i do it and how ??? and will it be applicable for all of them or a certain child job???

How to send parameters from child to parent... ???

 

Thanks,

Ankit

Employee

Re: Batch Processing/Batch mode in talend

Hi Ankit,

 

    Please refer the below link for the difference between multi threading and parallellization.

 

https://community.talend.com/t5/Design-and-Development/what-are-the-differences-between-Multi-thread...

 

     Regarding your next query, the parallel option do not restrict the job flow to any order. 

 

      For your last query regarding context variables, below articles will help you.

 

https://community.talend.com/t5/Design-and-Development/Passing-a-value-from-a-parent-Job-to-a-child-...

 

https://community.talend.com/t5/Design-and-Development/Returning-a-value-from-a-child-Job-to-the-par...

 

We are deviating a lot from original query. So I believe we should close the current topic and create another post for queries related to other topics.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now