API parallel requests

Six Stars

API parallel requests

Hi,

 

I'm looking for a solution to send concurrent requests to an API (preferably with one job only) but I'm not sure if this is possible in Talend at all. We are using Talend Open Studio.

It looks like the TRestClient component isn't capable to do this.

 

Best,

Kai 


Accepted Solutions
Community Manager

Re: API parallel requests

This will be tough without the subscription product. You could try putting your API calls inside child jobs, passing your parameters to the child jobs using context variables, and ticking "use an independent process to run subjob". This will potentially allow them to run in parallel. It's more of a hack than a solution I'm afraid. Parallelisation is one of the pieces of functionality that is made easier with the subscription product. You should consider how many parallel calls you wish to make at one time though. A rule of thumb would be n-1 where n is the number of cores running your job.


All Replies
Community Manager

Re: API parallel requests

This will be tough without the subscription product. You could try putting your API calls inside child jobs, passing your parameters to the child jobs using context variables, and ticking "use an independent process to run subjob". This will potentially allow them to run in parallel. It's more of a hack than a solution I'm afraid. Parallelisation is one of the pieces of functionality that is made easier with the subscription product. You should consider how many parallel calls you wish to make at one time though. A rule of thumb would be n-1 where n is the number of cores running your job.

Six Stars

Re: API parallel requests

Hi @rhall_2_0 ,

thanks for your reply, however it doesn't really solve our issue, but I will try to do this with several subjobs.

Best,

Kai

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads