I've created a job that returns JSON records from a GET REST endpoint, from 'system A', and map all fields, including an 'system A' ID for each record from that system, and use tWriteJSONFields to convert into another JSON schema format which is Posted to another REST endpoint, of another 'system B', which returns a 'system B' ID's, for the newly created records.
How do I match up the'system 'A' input data + 'system 'A' ID with the response data from 'System B' + 'System B' ID into a CSV file.
I tried assigning the 'System A' ID to a context var in tJavaRow, but the tRestClient uses the last stored 'System A' ID from the last assigned context var from the last iteration of the flow?
I'm not sure precisely what you are asking, but I *think* I kind of get the rough idea. I think you are asking about a way of ensuring you can take the data from one Rest service response, store it, send it to a second Rest service and link the first response record with the record returned by the second Rest service. Is that about right?
If so, there is an easy way of doing this. What you need to do is keep your job the same up until the record is returned from the first Rest service. Then use a tFlowToIterate which will allow you to, for every record returned from the first Rest service, carry out the logic for the second Rest service and then join the data. The reason for this is that the tFlowToIterate essentially pauses the preceding components until the rest of the flow is complete. So your records stored (in the globalMap by the tFlowToIterate) for the first Rest service are kept static for each iteration. There is an iteration of everything after the tFlowToIterate for every record returned by the first Rest service.
Thanks for this rhall_2_0. Your assumption is correct, I'm basically taking records from one system endpoint and mapping fields to post into another system via their respective endpoints
tRestClient ----- GET JSON response ---- tExtractJSON ----tMap ------(insert response fields into DB) --------tWriteJSON------
tRestClient-----POST JSON ---- tExtractJSON (response I'd of new record insert into DB match with records from lst tRest response)
Could you please give me a better understanding of where I would place tFlowToIterate in the above data flow.
Thanks for your help.
Looking at your flow I would suggest placing the tFlowToIterate after your tExtractJSON (after the first tRestClient). As a slight change to what I suggested before, add a tFixedFlowInput via Iterate link after the tFlowToIterate. Your column data will be added to the globalMap. You will need to retrieve this data and output it in the tFixedFlowInput component. To do this, copy the schema of the tExtractJSON schema (output schema) and paste it in the tFixedFlowInput. Then you will need to retrieve the data for each column from the globalMap. To do that, you need to be aware of the row name, the column names and the types (classes). For example, if you have a column called "name" which is a String and the row feeding the tFlowToIterate is called "row21", the globalMap code you would use to retrieve this value for the tFixefFlowInput "name" column would be.....
In the following flow, you can use the above syntax to retrieve the values stored in the globalMap via the tFlowToIterate.
I think I understand what your suggesting which is to store all the fields, on each iteration, into a globalMap, but I'm already storing these as rows in a DB, after tExtractJSON, from lst tRestClient, but I'm having trouble with updating DB, with the response ID's from the second tRestClient, as both Id's are unique to the system that the same record belongs to, I.e I'm transferring employee records from one system and creating them in another.
How would I then match the responses from the second tRestClient responses (newly created ids) and update their respective records in the DB, with the new id, that where inserted after the lst tRestClient?
Hope this makes more sense.
You need a common ID to do this. I thought your issue was that your common ID was coming from your first Rest call and that you couldn't use it after your second Rest call. Sorry, my mistake there.
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.