Batch Processing/Batch mode in talend

Eight Stars

Batch Processing/Batch mode in talend

Hi,

How does Talend handle Batch processing?

Does it generate seperate shell scripts with .bat extensions ?? and where do these .bat scripts get stored and executed??

How is it different from Linux processing?

In almost all Db components, we have enable parallel execution --> if enabled, how does it work??? does it split the records/data into child threads.??

 

Thanks,

Ankit.


Accepted Solutions
Highlighted
Employee

Re: Batch Processing/Batch mode in talend

Hi Ankit,

 

    Please refer the below link for the difference between multi threading and parallellization.

 

https://community.talend.com/t5/Design-and-Development/what-are-the-differences-between-Multi-thread...

 

     Regarding your next query, the parallel option do not restrict the job flow to any order. 

 

      For your last query regarding context variables, below articles will help you.

 

https://community.talend.com/t5/Design-and-Development/Passing-a-value-from-a-parent-Job-to-a-child-...

 

https://community.talend.com/t5/Design-and-Development/Returning-a-value-from-a-child-Job-to-the-par...

 

We are deviating a lot from original query. So I believe we should close the current topic and create another post for queries related to other topics.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

 


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)

All Replies
Employee

Re: Batch Processing/Batch mode in talend

Hi Ankit,

 

      There are lot of queries with different subjects. So lets take one by one.

 

Batch processing is the mode by which you are planning to process data in batch manner like files or DB delta data etc. The other mode is real time processing where we normally employs techniques like web services where response will be instant (or near instant).

Now, you can process batch processing jobs in multiple methods. Running Talend jobs through .sh/.bat files is just one way of processing those files. You can also run these files through TAC if you are using a subscription product. The new version of Talend even allows to create docker files. So it is a method of execution of these jobs. You can refer the details about the various methods from below link.

 

https://help.talend.com/reader/GQMv9RekYnlXzw3tanRDdA/jjIXcdA8eKKSN4YJe17FSg

 

Coming to third query, when you say about Linux processing, I guess you are talking about running the jobs through shell scripts. So it is a sub category of above method where Talend jobs are embedded in Linux shell scripts and you can schedule them through crontab. In Linux itself, if you have setup your TAC installation over a Linux machine, you can run your Talend jobs from there too. 

 

For your query related to DB components, Sabrina has already answered it in another post. Please refer the details below.

 

https://community.talend.com/t5/Design-and-Development/Parallel-Execution/td-p/33850

 

Hope you got answers for all your queries :-)

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

 


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)
Eight Stars

Re: Batch Processing/Batch mode in talend

Hi @nikhilthampi ,

Can you pls clear one final query of mine...

How is Multi-threading execution different from the Regular Parallel Execution??Does it depend on Job design ??

Another query is for example - i have five child jobs(sub jobs), i havent used tparallelize component..but how exactly the execution order commence...does it depend on least number of data which is to moved or complexity of job design etc....???

Another query is for example - i have 1 master comprising of 5 subjobs... now if i want pass a parameter from parent job to child can i do it and how ??? and will it be applicable for all of them or a certain child job???

How to send parameters from child to parent... ???

 

Thanks,

Ankit

Highlighted
Employee

Re: Batch Processing/Batch mode in talend

Hi Ankit,

 

    Please refer the below link for the difference between multi threading and parallellization.

 

https://community.talend.com/t5/Design-and-Development/what-are-the-differences-between-Multi-thread...

 

     Regarding your next query, the parallel option do not restrict the job flow to any order. 

 

      For your last query regarding context variables, below articles will help you.

 

https://community.talend.com/t5/Design-and-Development/Passing-a-value-from-a-parent-Job-to-a-child-...

 

https://community.talend.com/t5/Design-and-Development/Returning-a-value-from-a-child-Job-to-the-par...

 

We are deviating a lot from original query. So I believe we should close the current topic and create another post for queries related to other topics.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

 


Warm Regards,
Nikhil Thampi
Please appreciate our members by giving Kudos for spending their time for your query. If your query is answered, please mark the topic as resolved :-)

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.