API parallel requests

Highlighted
Six Stars

API parallel requests

Hi,

 

I'm looking for a solution to send concurrent requests to an API (preferably with one job only) but I'm not sure if this is possible in Talend at all. We are using Talend Open Studio.

It looks like the TRestClient component isn't capable to do this.

 

Best,

Kai 


Accepted Solutions
Community Manager

Re: API parallel requests

This will be tough without the subscription product. You could try putting your API calls inside child jobs, passing your parameters to the child jobs using context variables, and ticking "use an independent process to run subjob". This will potentially allow them to run in parallel. It's more of a hack than a solution I'm afraid. Parallelisation is one of the pieces of functionality that is made easier with the subscription product. You should consider how many parallel calls you wish to make at one time though. A rule of thumb would be n-1 where n is the number of cores running your job.

View solution in original post


All Replies
Community Manager

Re: API parallel requests

This will be tough without the subscription product. You could try putting your API calls inside child jobs, passing your parameters to the child jobs using context variables, and ticking "use an independent process to run subjob". This will potentially allow them to run in parallel. It's more of a hack than a solution I'm afraid. Parallelisation is one of the pieces of functionality that is made easier with the subscription product. You should consider how many parallel calls you wish to make at one time though. A rule of thumb would be n-1 where n is the number of cores running your job.

View solution in original post

Six Stars

Re: API parallel requests

Hi @rhall_2_0 ,

thanks for your reply, however it doesn't really solve our issue, but I will try to do this with several subjobs.

Best,

Kai

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog