tSOAP - access the response header

Six Stars fmh
Six Stars

tSOAP - access the response header

Hello,

i have been searching around now for quite a time, I want to communicate with a SOAP Webservice where the process is the following:

 

  1. connect with the Webservice via Login Credentials sent in the body part
  2. receive a Cookie from the header of the response to the login message
  3. use the received cookie for further requests on the webservice

This is a quite common scenario.

 

Component options which i have tested:

- tSOAP component - no chance to get the HTTP header information out of that

- tWebservice component - same.

- tHttpRequest - no chance to get the HTTP header out of the response, only the body part - WTF is this component else for instead of raw HTTP requests?!?

- tESBConsumer component with hacks to get to the HTTP header - does somehow not work for me (see https://community.talend.com/t5/Design-and-Development/resolved-SOAP-header-in-tJavaRow/m-p/78503#M3...)

 

Can someone suggest a solution? Reading out a cookie from a HTTP response Header is not that uncommon and i think Talend should support this kind of stuff out of the box, tell me when i'm not right. My alternative currently is to just use Java Code for building a component which should already exist?!?


Accepted Solutions
Ten Stars

Re: tSOAP - access the response header

Ooooh dear..... well ... uhhhm....  tSystem and cUrl

 

Make sure the cUrl is called like this, assuming your on some sort unix/linux distro, so in tSystem:

 

new String[]{"bash","-c","curl -k -L -v -c "+((String)globalMap.get("cookiesFolder"))+"/cookie.txt https://somefreakingurl/login -d 'login=login&username="+context.Usr+"&password="+context.Pwd+"'" }

search internet for tSystem and bash array because of loading the same bash environment when you start your talend job from command line. 

I quit using all of the Talend components... cUrl exactly because of what  you experienced.

 

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

Hello rhall_2_0,

I now managed to get the tESBConsumer component working. Somehow my Talend Installation won't let me automatically produce the XML Schemas from the WSDL file when I create a Service in the Context and import the WSDL file there. I created the XML Schemas on my own and then changed it in the tXMLMap component again. Here's what i've done in screenshots:

 

solution_0_flow.pngthe flowsolution_1_tXMLMap.pngthe XMLMap component to generate the SOAP Body Contentsolution_2_tESBConsumer_advanced_settings.pngThis is my advanced settings where I put SOAP settings in the Headersolution_3_tExtractXMLField.pngExtracting the Login Response (true/false)solution_4_tMap.pngMapping the Response (maybe i should rewrite the error condition just as a negation of the true response)

 

Now that I got the response, I am stuck at extracting the response's header.

I added a tJava Component with the code below and i'm just getting a nullPointerException.

 

Advanced Settings

import java.util.List;
import java.util.Map;
import java.util.Iterator;

Code

System.out.println("Get Cookie from Header");
try {
	java.util.Map<String,java.util.List<String>> headers =((java.util.Map<String,java.util.List<String>>)globalMap.get("tESBConsumer_3_HEADERS"));

java.util.Iterator<String> it = headers.keySet().iterator();
	while(it.hasNext()){
		System.out.println(it.next());
		// todo: cookie processing
		// check if line has the cookie in it
		// if yes, write the cookie to a context variable
	} 
} catch (Exception nullPointerException) {
	// TODO: handle exception
	System.out.println("nullPointerException");
}

 

The tESBConsumer Headers should be a Map with a String and a String List, am I doing anything wrong, e.g. the Java Code is executed before the tESBConsumer has received the Headers and the value is avaiable in the globalMap?

 

tESBConsumer_3_HEADERS.png

Sixteen Stars

Re: tSOAP - access the response header

This is down to a bug in the documentation. I should have known as I have actually hit this before. The globalMap variable is actually tESBConsumer_3_HTTP_HEADERS. Replace the code with the code below....

 

System.out.println("Get Cookie from Header");
try {
	java.util.Map<String,java.util.List<String>> headers =((java.util.Map<String,java.util.List<String>>)globalMap.get("tESBConsumer_3_HTTP_HEADERS"));

java.util.Iterator<String> it = headers.keySet().iterator();
	while(it.hasNext()){
		System.out.println(it.next());
		// todo: cookie processing
		// check if line has the cookie in it
		// if yes, write the cookie to a context variable
	} 
} catch (Exception nullPointerException) {
	// TODO: handle exception
	System.out.println("nullPointerException");
}

 


All Replies
Ten Stars

Re: tSOAP - access the response header

Ooooh dear..... well ... uhhhm....  tSystem and cUrl

 

Make sure the cUrl is called like this, assuming your on some sort unix/linux distro, so in tSystem:

 

new String[]{"bash","-c","curl -k -L -v -c "+((String)globalMap.get("cookiesFolder"))+"/cookie.txt https://somefreakingurl/login -d 'login=login&username="+context.Usr+"&password="+context.Pwd+"'" }

search internet for tSystem and bash array because of loading the same bash environment when you start your talend job from command line. 

I quit using all of the Talend components... cUrl exactly because of what  you experienced.

 

Sixteen Stars

Re: tSOAP - access the response header

This isn't a hack, it is just using the HEADERS globalMap value returned by the tESBConsumer component (if headers are returned). The code below should simply list the key values to all headers returned. You should be able to progress from here. This assumes that your tESBConsumer is tESBConsumer_1.

 

java.util.Map<String,java.util.List<String>> headers = ((java.util.Map<String,java.util.List<String>>)globalMap.get("tESBConsumer_1_HEADERS"));

java.util.Iterator<String> it = headers.keySet().iterator();

while(it.hasNext()){
	System.out.print(it.next());
} 

  

Highlighted
Six Stars fmh
Six Stars

Re: tSOAP - access the response header

 

Hello Dijke,

 

thanks for your reply. I got the tSystem component running in general, but with my curl string (which works fine on the command line) the tSystem component can't handle whitespaces somehow. That is the String i have in my tSystem component:

 

 

"curl -k -L -s -c cookie.txt --header \"Content-Type: text/xml;charset=UTF-8\" --header \"SOAPAction: login\" --data @login.xml https://theSoapEndpointIamCalling.com"

 

so this is my error message:

 

curl: no URL specified!
curl: try 'curl --help' or 'curl --manual' for more information
/bin/bash: login --data @login.xml https://theSoapEndpointIamCalling.com: No such file or directory
|curl: no URL specified!

it looks like the call is cutted after SOAPAction: and then "login" is called with parameters, which is not the expected behaviour.

 

 

 

Do you have a hint for getting this working? My stupid simple solution would be just calling a bash script which is calling the curl command, but i would like to avoid having additional scripts.

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

Hello rhall_2_0,

 

i tried the tESBConsumer component solution, but until now i did not managed to get the correct SOAP message for the request.

My SOAP request looks like the following:

 

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:Myprovider_myreport">
   <soapenv:Header/>
   <soapenv:Body>
      <urn:login>
         <user>myuser</user>
         <pass>mypassword</pass>
         <network>mynetwork</network>
      </urn:login>
   </soapenv:Body>
</soapenv:Envelope>

 

I am just trying to load the XML file with the content from above with a tFileInputXML component, loop over the soap envelope and then push the content from the body in the payload, but i only receive errors from the API stating login errors (when cUrl'ing with that file, everything works fine and i receive a login OK response and a cookie).

 

tESBConsumer_1.pngtESBConsumer_2.png

Is there something wrong with my input for the tFileInputXML?

 

btw, I'm using Talend Real-time Big Data Platform 7. I have quite a bunch of similar jobs and was hoping that the tSOAP component is just "complete" in terms of usage, instead I now must go with an tESBConsumer workaround or a tSystem curl workaround - I am already thinking of coding an addition to the tSOAP component which fullfils my needs to access the responding HTTP header for cookie extraction...

 

Thanks already for all the people replied here, the Talend community is quite active now in comparsion to a few years ago Smiley Happy I hope to be able to give something back, I'll scan the incoming questions and will document / explain my solutions here as complete as I can.

Ten Stars

Re: tSOAP - access the response header

No idea... 

Did you call cUrl using the array call like my example, or just directly the command?

Seems like they (tSystem vs bash prompt ) differ.

 

In other examples I've seen the soapaction within quotes, like:

\"SOAPAction: "urn:loginRequested"\"

Maybe play around with single quotes... any documentation on your wsdl part, because your soapaction could also be a full url?

 

Sixteen Stars

Re: tSOAP - access the response header

OK, first of all you need to see that you are getting the correct data from the tFileInputXML. To do this, remove the tESBConsumer and connect a tLogRow to the tFileInputXML. This will print out anything retrieved from the file to the output. I suspect that you are either not retrieving anything or you may not be retrieving the correct section of the XML. Since your XPath seems to show you want the Body, I actually think you are probably struggling with missing namespace information in your XPaths. There is actually an option on the tFileInputXML's Advanced Settings to ignore namespaces. 

 

Play around with those settings with the tLogRow connected. Don't connect the tESBConsumer until you are getting the XML snippet you expect

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

Hello rhall_2_0,

i tried to get the payload (and additionally the header, which i do not need - i just need the header of the server response with the Set-Cookie attribute for further API calls) as debug output as seen below. I expected an <urn:login> part, but i just got an <login> part. I used the "Namespace Ignore" option in the additional settings and used the Encoding Type: UTF-8 as the XML file is UTF-8-encoded. Result below.

 

tESBConsumer_3_debug.png

 

Without having the "Namespace Ignore" option on, i get a result which is nearly what i wanted - just that i have the additional attribute xmlns:run="urn:blablabla" at the <urn:login> - that is okay (see below).

tESBConsumer_4_debug.png

 

But I got a leading <?xml version="1.0" encoding="UTF-8"?> which i do not need and which results in errors of the API. I tried that log output directly as the body part with SOAP UI and it gives me Encoding errors; when i remove the <?xml ...?> tag it works fine and gives me a successful login response as expected.

 

Final (?) question now: How do i remove the <?xml version="1.0" encoding="UTF-8"?> Tag from the payload stream (type document) before it enters the tESBConsumer component?

Sixteen Stars

Re: tSOAP - access the response header

The XML header is added automatically to the XML document to make it XML compliant. Can you tell me whether your payload column (being returned by the tFileInputXML) is a Document or a String? It should be a Document. The ESBConsumer (assuming it has been configured appropriately via the "Service Configuration" button) should be able to handle the column if it is a Document.

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

The Output is a Document. But as a document with a leading <?xml ...?> definition the API will not accept the request and give me errors. My idea was now to use a simple tReplace component to find and delete the <?xml version="1.0" encoding="UTF-8"?> String from the Document, but as the tReplace does not accept Documents as Input, it gives me errors. Is there an easy way or do I need to build a document to string -> replace -> string to document construction for that, and when i need to do that, which components can i use to transform document to string and vice versa?

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

can i somehow just use a tJavaRow or something like that to just remove the string by a simple regex? My brain hurts when thinking of this overengineered solution for just making components work for standard needs. Dear Talend original component coders, I am not pleased by your solution of handling SOAP! Smiley Wink The funny thing is, that i'm replacing a Pentaho ETL job with a Talend job here - and in Pentaho the solution was already a nearly complete rebuild of a SOAP communication componente for the request.
Sixteen Stars

Re: tSOAP - access the response header

OK, here is another experiment to show you what I mean. Where you had a tLogRow, replace it with a tFileOutputXML (leaving all of the defaults). Run the job and open the output file. You will notice your XML wrapped in other elements (your column name and one called row with an outer element of root). At the top you will see the XML header. This is what will happen with the tESBConsumer. You do not need to worry about the header, Your error is elsewhere. IF you tested the XML with the header using another tool, yes you will see an error. Talend will not send the body with the XML header.

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

Hello rhall_2_0,

 

i tried with a tFileOutputXML with all defaults and this is my resulting file:

tESBConsumer_7_tFileOutputXML.png

 

if i just use whats in the payload for the SOAP body part in an external tool like SOAP UI, it gets accepted (see below). I simplyfied it now for having only the payload, as i do not need to send any information via the header. As you can see on the last part of the screens here, the correct action is selected.

tESBConsumer_5.png

When combining this with just the tESBConsumer part as defined in my other messages here, i get the following errors:

tESBConsumer_6_Error.png

Do i need to send SOAP headers additionally somehow? My example request has an empty soapenv:header part but is using the urn:xxxxxx_xxxx#login action, the same which I selected in the tESBConsumer Component as seen above. 

 

Sixteen Stars

Re: tSOAP - access the response header

Your example that errors in SOAP UI is NOT what Talend is doing. The XML header is ignored. If it were not, absolutely no jobs that use the tESBCOnsumer would work. I have several working here.

 

The error you are getting from Talend is telling you there is an encoding violation with your data. SOAP UI is far more forgiving than practically any other tool. For us to be able to help we will need to see the WSDL and the data you were sending.

 

 

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

This is parts of the WSDL where i replaced the service provider name with "Provider" and the exact WSDL service with "serviceReport". I excluded all the other topics from the WSDL here...

wsdl_extract.png

Sixteen Stars

Re: tSOAP - access the response header

I can't see what you are supplying, but my suspicion is that the Network element may be causing this issue. The WSDL has it requiring an int, but in your "Expected" example from earlier you have it set to MYNETWORK. Are you passing a String or an int? 

 

Unfortunately web services are notoriously difficult to debug since you seldom have control over both ends (client and server). 

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

I am passing an integer. The contents of the body when I copy it from the log or from the experimental xml output file are accepted by the API and return a successful login result, when I do the request via SOAP UI. Same goes for just submitting the XML file (which I read in with the tFileInputXML component) directly via curl. My assumption is that the tESBConsumer component somehow messes up the SOAP envelope structure. I will use Wireshark to see what exactly Talend is sending out to the server, I already thought about building a micro nodejs server to replace the api endpoint and show me the raw request...
Sixteen Stars

Re: tSOAP - access the response header

There is something else to try. You should really build the XML with Talend. If you follow these steps, you can import the request schema from your WSDL and use it with a tXMLMap to build it before passing it to the tESBConsumer.

 

1) Find the Services branch in your project's tree structure, right click on it and select "Create Service". Give it a name, and click Next. Then select "Import WSDL". Put your WSDL url in the box that appears (remember it must end with ?WSDL). Click Finish.

2) Now go to your Metadata, then expand File XML. You should see a folder within there which is related to your SOAP service. Inside, it will hold both the Request and Response schemas for your service. I assume you have many actions, so there may be a few of these.

3) Now add tFixedFlowInput component to your job. Make it the start component. Add appropriate typed columns to hold the data you need to send.

4) Now add a tXMLMap component and join it to the tFixedFlowInput. Open the tXMLMap and add an output table. In that, add a "payload" column of type Document. Right click on the payload column and select "Import from repository". Now find your Request schema. When you have selected it, your Request schema will appear. 

5) Connect up your input columns to their respective output elements.

6) Connect your tXMLMap to a tLogRow and run the job. Take a look at the format of the XML. You can also test this in SOAP UI (remember to remove the XML header for SOAP UI....Talend does this for you). If it works, then remove the tLogRow and connect your configured tESBConsumer.

 

If you follow this process, it should eliminate any errors that may have crept in by dicing and splicing XML. 

Sixteen Stars

Re: tSOAP - access the response header

Did this work for you?

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

Hello rhall_2_0,

I now managed to get the tESBConsumer component working. Somehow my Talend Installation won't let me automatically produce the XML Schemas from the WSDL file when I create a Service in the Context and import the WSDL file there. I created the XML Schemas on my own and then changed it in the tXMLMap component again. Here's what i've done in screenshots:

 

solution_0_flow.pngthe flowsolution_1_tXMLMap.pngthe XMLMap component to generate the SOAP Body Contentsolution_2_tESBConsumer_advanced_settings.pngThis is my advanced settings where I put SOAP settings in the Headersolution_3_tExtractXMLField.pngExtracting the Login Response (true/false)solution_4_tMap.pngMapping the Response (maybe i should rewrite the error condition just as a negation of the true response)

 

Now that I got the response, I am stuck at extracting the response's header.

I added a tJava Component with the code below and i'm just getting a nullPointerException.

 

Advanced Settings

import java.util.List;
import java.util.Map;
import java.util.Iterator;

Code

System.out.println("Get Cookie from Header");
try {
	java.util.Map<String,java.util.List<String>> headers =((java.util.Map<String,java.util.List<String>>)globalMap.get("tESBConsumer_3_HEADERS"));

java.util.Iterator<String> it = headers.keySet().iterator();
	while(it.hasNext()){
		System.out.println(it.next());
		// todo: cookie processing
		// check if line has the cookie in it
		// if yes, write the cookie to a context variable
	} 
} catch (Exception nullPointerException) {
	// TODO: handle exception
	System.out.println("nullPointerException");
}

 

The tESBConsumer Headers should be a Map with a String and a String List, am I doing anything wrong, e.g. the Java Code is executed before the tESBConsumer has received the Headers and the value is avaiable in the globalMap?

 

tESBConsumer_3_HEADERS.png

Sixteen Stars

Re: tSOAP - access the response header

The code looks fine. Can you try moving the tJava to connect to the tESBConsumer_3 using the OnComponentOK link. This is just a test. Unfortunately I cannot try anything out as I don't have your service available to me.

 

Regarding Talend not being able to build the schema from the WSDL, that is odd but can be caused by WSDLs that refer to XSDs. 

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

I tried it with OnComponentOk but i'm still getting a NullpointerException
Sixteen Stars

Re: tSOAP - access the response header

This is down to a bug in the documentation. I should have known as I have actually hit this before. The globalMap variable is actually tESBConsumer_3_HTTP_HEADERS. Replace the code with the code below....

 

System.out.println("Get Cookie from Header");
try {
	java.util.Map<String,java.util.List<String>> headers =((java.util.Map<String,java.util.List<String>>)globalMap.get("tESBConsumer_3_HTTP_HEADERS"));

java.util.Iterator<String> it = headers.keySet().iterator();
	while(it.hasNext()){
		System.out.println(it.next());
		// todo: cookie processing
		// check if line has the cookie in it
		// if yes, write the cookie to a context variable
	} 
} catch (Exception nullPointerException) {
	// TODO: handle exception
	System.out.println("nullPointerException");
}

 

Six Stars fmh
Six Stars

Re: tSOAP - access the response header

Hello rhall_2_0,

 

i just found out the same; the autocomplete gives me _HEADERS while the documentation says _HTTP_HEADERS.

I opened an issue in the bug tracker for that, https://jira.talendforge.org/browse/TESB-23531

 

thank you very very much for your help so far,

best regards,

Finn