[resolved] How to pass data through a job ?

One Star

[resolved] How to pass data through a job ?

Hello,
I'm new to Talend and I'm trying to achieve my first job for a network project.
Unfortunately I'm stuck at the moment because of my lack of knowledge of the software and its components.
Explanations
I'm collecting data from a network management console API (Cisco Prime Infrastructure) and I need to proceed with 4 API query to get all the necessary data : device id, ip address, interfaces and their descriptions. When the job has gathered everything together, the data are sorted and a last API query is made to the management console.
But the problem is that I don't know how to "link" each rows to the others through the different API calls. I made a pattern to make it clearer :

By now, I know how to get the data and parse them in order to get what I want from the XML files but I can't find how to get the data to "follow" through the job and be linked together.
For each device id should be linked an ip address, a list of its interfaces, the intefaces' descriptions... But how to proceed ?
Thank you.
Cardiganup.

Accepted Solutions
Community Manager

Re: [resolved] How to pass data through a job ?

Hi  
If I understand your requirement well, you just need to add a tFlowToIterate between Parse device_no and Get interfaces to iterate each device no. eg:
...Parse device_no--row16--tFlowToIterate--iterate--Get interfaces--main--parse interfaces--main--tMap--tLogRow
change the URL in step 3 to:
3) "Get interfaces" Query : " + (String)globalMap.get("row16.device_no")
//Assuming the data type of device_no is string, if it is integer type in your case, change it to 
"Get interfaces" Query : " + (Integer)globalMap.get("row16.device_no")
on tMap, add a new column called device_no in the output table, and set its expression as:
(String)globalMap.get("row16.device_no")

Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business

All Replies
Community Manager

Re: [resolved] How to pass data through a job ?

Hi 
tRestClient requires a Document as input request or returns a Document as response, so you can use tXMLMap to extract data from the response or build the Document as input request. For example:
tRestClient1--tXMLMap--tRestClient2--tXMLMap...
Refer to the component manual and learn tXMLMap component. 
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] How to pass data through a job ?

Hi,
Thanks for your answer Shong.
I managed to parse the XML responses with a tExtractXMLFields, I can't get tXMLMap to work.
Here's how I did it :

1) "Get devices" Query : "
2) "Parse device_no" parsing the tRESTClient response.
3) "Get interfaces" Query : " + row16.device_no
4) "Parse interface" parsing the tRESTClient response.
I get the results I expect BUT I don't know how to link the devices' ID and the interfaces, how to get that :



So to each device id is linked its interfaces, ip, ... Is there a way to make like individual arrays to store the data for each device along the job?
Thank you.
Cardiganup.
Community Manager

Re: [resolved] How to pass data through a job ?

Hi  
If I understand your requirement well, you just need to add a tFlowToIterate between Parse device_no and Get interfaces to iterate each device no. eg:
...Parse device_no--row16--tFlowToIterate--iterate--Get interfaces--main--parse interfaces--main--tMap--tLogRow
change the URL in step 3 to:
3) "Get interfaces" Query : " + (String)globalMap.get("row16.device_no")
//Assuming the data type of device_no is string, if it is integer type in your case, change it to 
"Get interfaces" Query : " + (Integer)globalMap.get("row16.device_no")
on tMap, add a new column called device_no in the output table, and set its expression as:
(String)globalMap.get("row16.device_no")

Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] How to pass data through a job ?

Hi Shong,
Many thanks ! I managed to make the job I wanted Smiley Happy
Unfortunately, it's not completely done yet because I'm facing another issue...
I tried to switch from my test local environnement to the "real" production environnement I'm supposed to work with, but the tRESTClient can't connect to the server. Let me explain :

Server : Set on the Local Area Network, access via HTTPS, self-signed certificate.
Job :

I added the certificate to the "cacerts" Java file and I set its directory in the tSetKeyStore.
The tRESTClient tries to connect to a url that looks like : https://<certificate-attached-domain>.
I checked "Use authentification" with a "Basic HTTP" type and filled in the credentials.
But it's not working :
 connecting to socket on port 3530
connected
Exception in component tRESTClient_1
disconnected
javax.ws.rs.NotAcceptableException: HTTP 406 Not Acceptable
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:495)
at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:897)
at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)
at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:428)
at org.apache.cxf.jaxrs.client.WebClient.get(WebClient.java:611)
at ciscoprime.cisco_prime_0_1.Cisco_Prime.tRESTClient_1Process(Cisco_Prime.java:656)
at ciscoprime.cisco_prime_0_1.Cisco_Prime.tSetKeystore_1Process(Cisco_Prime.java:389)
at ciscoprime.cisco_prime_0_1.Cisco_Prime.runJobInTOS(Cisco_Prime.java:929)
at ciscoprime.cisco_prime_0_1.Cisco_Prime.main(Cisco_Prime.java:774)

And if I check "Follow redirects" and allow redirections to different hosts, here's what I get :
 connecting to socket on port 3349
connected
: org.apache.cxf.transport.http.HTTPConduit - WWW-Authenticate response header is not set
Exception in component tRESTClient_1
javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:495)
at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:897)
at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)
at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:428)
at org.apache.cxf.jaxrs.client.WebClient.get(WebClient.java:611)
at ciscoprime.cisco_prime_0_1.Cisco_Prime.tRESTClient_1Process(Cisco_Prime.java:662)
at ciscoprime.cisco_prime_0_1.Cisco_Prime.tSetKeystore_1Process(Cisco_Prime.java:389)
at ciscoprime.cisco_prime_0_1.Cisco_Prime.runJobInTOS(Cisco_Prime.java:935)
at ciscoprime.cisco_prime_0_1.Cisco_Prime.main(Cisco_Prime.java:780)
disconnected

I also tried to pass the authentification credentials through a Authorization header but it is not working as well.
Have you ever seen this behavior with a tRESTClient?
PS: Prior to using "tRESTClient", I was working with "tREST" components and it was going well.
Thanks.
Cardiganup.
One Star

Re: [resolved] How to pass data through a job ?

I solved the problem !
Apparently, the server didn't like to be forced to send XML back so I put "Any" in the type list :

Cardiganup.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

An API-First Approach to Modernizing Applications

Learn how to use an API-First Approach to Modernize your Applications

Watch Now

Talend API Designer – Technical Overview

Take a look at this technical overview video of Talend API Designer

Watch Now

Getting Started with APIs

Find out how to get started with APIs

Read