Merging Data from conditional REST call into flow

Four Stars

Merging Data from conditional REST call into flow

Hi, I'm new at Talend (and ETL), so apologies if I'm trying to do something that doesn't make sense.

I'm trying to conditionally add data from a REST call for some of the records I'm processing.

This is what I'm trying to achieve, with the red line being the lookup I'd like to add, but doesn't seem to be possible:

TalendFlow.png

I'm conditionally making a REST call for some of the records from the first iterate, and then I want to create a combine flow of the original data plus the results from the rest call (with those columns being null for the records which didn't make the REST call) - effectively making the output from the REST call a lookup on the main flow, without having to store the results to a table or file.

Given that the output from the tExtractXML is a flow per iteration (where I really need to add some logic to retrieve the first match from the XML), I think it makes sense that I wouldn't just be able to combine that with a flow for the entire result set, but I'm stuck trying to figure out how to do this via the Talend components rather than just making the REST call part of a Java function and hiding the details away in something like tJavaRow.

 

Thanks,

Andrew

Tags (1)

Accepted Solutions
Seven Stars

Re: Merging Data from conditional REST call into flow

Hi @AndrewMay you can store the data from tJavaRow_1 in tHashOutput component and then you can use tHashInput component with your tMap_3

 

Here is info showing where to find tHash... component https://community.talend.com/t5/Migration-Configuration-and/Where-are-the-tHashInput-and-tHashOutput...

 

and here you can see how to use it: https://www.youtube.com/watch?v=V5zqUtzQbBk

 

Don't forget to give kudos when a reply is helpful and when your query is answered, please mark the topic as resolved.

DataTeam.pl


All Replies
Seven Stars

Re: Merging Data from conditional REST call into flow

Hi @AndrewMay you can store the data from tJavaRow_1 in tHashOutput component and then you can use tHashInput component with your tMap_3

 

Here is info showing where to find tHash... component https://community.talend.com/t5/Migration-Configuration-and/Where-are-the-tHashInput-and-tHashOutput...

 

and here you can see how to use it: https://www.youtube.com/watch?v=V5zqUtzQbBk

 

Don't forget to give kudos when a reply is helpful and when your query is answered, please mark the topic as resolved.

DataTeam.pl

Four Stars

Re: Merging Data from conditional REST call into flow

Thanks, I think that puts me on the right track (and I wouldn't have found where those components were without the link).

I hit a few snags when attempting to implement this, but I think that's all due to the order things are initialized within a subjob (i.e. the lookups are done early) - I think I can work around that by doing my REST calls in a subjob before I use the results as a lookup.

Seven Stars

Re: Merging Data from conditional REST call into flow

@AndrewMay do you mean subjob or child job?

 

Subjob is within a job (a few components joined together).

Child job is a separated job you can run within another job by tRun component.

 

Sometimes we create complex separated job and we run it within another job (parent job)

If you want to use child job you will need to use tBufferInput and tBufferOutput components to get data from child job instead of tHash... components which you can use within a job to get data from sub job..

DataTeam.pl

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog