[resolved] Talend job running as a Web Service - input parameters

Highlighted
One Star

[resolved] Talend job running as a Web Service - input parameters

I have a concern about input parameters for a Talend job that is deployed as a web service.
From the documentation I read it looks like I will need to design all the Web Services (jobs) with context variables that I will be passing to the job for processing
I could have jobs like

GetCustomerById(ID) - would have context.ID
http://localhost:8080//export_job/services/export_job3?method=runJob&arg1=--context_param%20ID=2.

CreateCustomer(FirstName, LastName) - would have context.FirstName and context.LastName
http://localhost:8080//export_job/services/export_job3?method=runJob&arg1=--context_param%20FirstName=F&arg2=--context_param%20LastName=L.

It looks quite simple


BUT here is my concern.... what if the URL is exceeding the URL max size, which might be 2000 characters? (just a guess, I don't know what is the real max size)

1. what if the input parameter is longer than a string of 2000 characters:
CreateCustomer(FirstName, LastName, Comment) - context.Comment as a very long long string
http://localhost:8080//export_job/services/export_job3?method=runJob&arg1=--context_param%20FirstName=F&arg2=--context_param%20LastName=L&arg3=--context_param%20Comment=veryLongComment.......

2. what if the list of parameters is too long and I would be using like 200 arguments:
CreateCustomer(FirstName, LastName, arg1, arg2, ...... arg200) - list of arguments is 200 and again there is a possibility of very long long URL
http://localhost:8080//export_job/services/export_job3?method=runJob&arg1=--context_param%20FirstName=F&arg2=--context_param%20LastName=L&arg3=--context_param%20arg3=arg3........&arg200=--context_param%20arg200=arg200

Do you have same concern, or do you use different approach to read the pass in the input parameters when exposing jobs through the Web services?

Cheers
Lubo
Community Manager

Re: [resolved] Talend job running as a Web Service - input parameters

Hi Lubo

I have never concern the long path of URL, don't know what's the error might get if the path is very long as you said, please show it to us if you get an error.
You have to specify the context parameter in the URL when the webservice is calling from browser, you can also call the webservice in Talend Studio with tWebservice component, and specify the context parameter in the table list, the parameters can be set at the runtime. Please read calling a webservcie in Talend Studio.


Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] Talend job running as a Web Service - input parameters

Thanks Shong.

In other words we can assume the URL size limitation might be just a WEB browser issue and not implementation itself.

I guess I will have to do a proof of concept jobs,
1. Web service which will take Long Parameter
2. Web service which will take 200 Parameters - I guess this will be obsolete as I could just have an Long String (if No 1 works) as XML with parameters passed in as elements/attributes instead of all the individual context variables.

And have it tested from
a) all browsers (might fail)
b) from other Talend job using tWebServiceInput (should be always OK)
c) our company UI (this is where we intend to use the web services)
We are planing to use the Talend Web services as our main Application Data Layer.
If c) is not working then panic!

Still don't know when I do this as first of all I will need a Tomcat access to deploy the web service.
I have no admin power and the company I work for has very strict policy about installed software and permissions.
Community Manager

Re: [resolved] Talend job running as a Web Service - input parameters

Hi
Yes, I am interesting in your test result on access webservice with a very long URL in browser. Keep posting if you have time/chance to test it.

Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] Talend job running as a Web Service - input parameters

Today I got my test for a long string deployed on Oracle WebLogic server and here are the results...


Test a)
I performed the test just as a COPY and PASTE into browser URL

I was able to pass string of 1914 characters into the context parameter in IE.
It looks like the IE is able to take maximum 2042 characters in the URL (address+parameter)

With Google Chrome the limitation might be something more than a few millions. I stopped after 2 millions as the PASTE took really long time plus the respond (which contained the input value) was taking too long to interpret. The longer string the longer response and I am not waiting longer than few seconds... 3 millions looks like too much, but that might be limit set by my PC and MY PATIENCE, not the browser.

Summary:
IE - URL of 2042 characters
Chrome - URL of more than 2 millions
due to company policy i was not able to test other browsers.

Test b)
Using tWebServiceInput , parameter is random string length incremented by 1 million each iteration,
After 23 million characters, I am getting Exception in component tWebServiceInput_1
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.lang.reflect.InvocationTargetException

Summary: 23 millions just fine, 24 millions started failing.
I don't know if this is due to my computer specification and perhaps a better spec computer might be able to process even higher numbers.

Test c)
This will have to be be up to somebody else, but in theory it should be same as Test 2.

Conclusion:
Test b shows it is possible to process really huge strings and in the real world scenarios the longest string will never exceed 10 thousand characters, so there is still a huge reserve in the limitation.
I am not sure if the test would be same on a Tomcat, but Oracle WebLogic is the company's choice.
Community Manager

Re: [resolved] Talend job running as a Web Service - input parameters

Hi lubod

Thank you a comprehensive testing and summary.Smiley Wink


Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] Talend job running as a Web Service - input parameters

Hi All,

I am exporting talend job as web service. While calling the web service url, i am trying to pass context parameter as argument. But i am not getting the expected result. web service is taking the default context parameter value which is specified inside talend studio.
Below mentioned is the url for web service


        

Thanks
Rajath M Y

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Have you checked out Talend’s 2019 Summer release yet?

Find out about Talend's 2019 Summer release

Blog

Talend Summer 2019 – What’s New?

Talend continues to revolutionize how businesses leverage speed and manage scale

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog