Six Stars

curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Hi,

I have to perform a cURL with Talend, and I would know if tREST or tRESTClient are useful for that, instead of a generic tSystem Component.

 

curl -X GET <url>/<path> -d '<json-data>'

My curl is aimed to perform a GET request, pass a "data" (json) parameter and retrieve a json response.
It needs also to be authenticated.


Well, speaking about authentication, tREST / tRESTClient are great, the work fine (tREST permith to specify headers good for authentication, and tRESTClient works perfectly because is fully configurable in terms of authentication method).

My problem is that my curl have to pass json data:
curl -X GET <url>/<path> -d '<json-data>'
and I don't know how to pass this data (body) since I have to perform a GET (not a POST).

Any help?

Thank you,
Lorenzo

 

12 REPLIES
Employee

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

If you pass the JSON Body as a payload document to tRestClient, does it not work?  You will pass the body as a row from a previous component.

 

Six Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Hi @iburtally

 

Unfortunately it doesn't work.

 

I put a tFixedFlowInput as previous (of tRESTClient) component, to pass the body, but the tRESTClient seems to ignore it.

Since the Input Schema of the tRESTClient has originally two fields, "body" of type Document and "string" of type string, I have tried to:

- fill the "string" input field with my json string body and leave empty the "body" field

- fill the "string" input field with my json string body and delete the "body" field

but both of the attempt failed (tRESTClient component does not consider the input field "string", when calling the REST server)

 

Any idea?

 

Six Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Mmmmm...

 

I noticed that when you perform a GET with tRESTClient component you are not able to specify the ContentType of your request (the field disappear).

That probably means the component isn't able to send a payload in GET, is it?

 

So, maybe the focus should be on my original question: how to perform a curl, in GET, specifing the "data" (-d or -data, for linux) parameter? 

For this purpose tSystem component is very unfriendly because any complex command need to be managed with the "Use Array Command" (tSystem Component option) that, unfortunately, split on spaces and my json payload is full of spaces :-(

 

 

Employee

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Normally, when you do a GET, you will pass query parameters.  I have not come across the ones that require a body data.  If it doesn't work, then yes, tSystem is your best bet right now.  Generally when passing JSON data like this, I have seen other services looking for a Base64 encoded string.  You encode your json as base64 and the service decode it, just to avoid that spaces problem etc.  Maybe you should try a URL Encode on the JSON data.

Eleven Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Use the HTTP Headers to set this on the Advanced Settings tab of the tRESTClient. The name will be "Content-type" and the value will be "application/json".

 

Rilhia Solutions
Six Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Hi @rhall_2_0

 

as you can see, the problem is not the content-type itself.

The problem is that when you use tRESTClient with HTTP Method = GET you have no way to specify a payload (the fields that comes from the Input Link, "body" and "string", seem to be ignored by the tRESTClient).

 

So, my question is:

is it possible to use tRESTClient to perform the following curl, with GET http method and with a specific "data"?

curl -X GET <url> -d "<json-payload>"

 

Thank you,

Lorenzo

Six Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Hi @iburtally

 

thank you for your answer. I'm sending request to an ElasticSearch service.

ElasticSearch, as most of the other Restful Services, require to use GET to get some data ("_search" api) and PUT, POST, ... to write some data.

And yes, when you search for some data you have to specify query-constraints and/or specific aggregation rules, you want Elasticsearch applies.

 

On the other hand, infact, the unix "curl" let you specify "--data" (or "-d") parameter also when you perform a GET request.

 

Thank you for your help. tSystem is also a little complicated to configure, I'm trying and I'm getting some trouble. There is not, unfortunately, a good documentation (with various use-cases) on Talend Components.

 

Lorenzo

Eleven Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

@Lorenzo I was answering your sub question about the Content-Type as @iburtally had answered your main question....or at least attempted to.

 

For your main question, maybe you can try using something like below....

 

http://Your_uri?data={your: "json"}

You will probably have to encode your JSON to be compatible with a URL.

 

I haven't tested this since I don't actually use GET REST services that require a body instead of params, but this *may* work.

Unfortunately with web services, each implementation is usually subtley different to the next. You cannot rely on ANY tool catering for all of the cases out of the box and there is usually some jiggery pokery involved in trying to get them to work. I am currently playing around with the Spotify API using OAuth authentication. I've used dozens of OAuth secured services, but this one is proving to be a real pain. 

 

Rilhia Solutions
Four Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Hello,

Please could Talend community can answer to this question ?

 How can we query elasticsearch from Talend, no way to do that on tRest or tRestCleient ?

My simple query is like that: curl -XGET "http://localhost:9200/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"multi_match" : {
"query": "VIEILLES",
"type": "best_fields",
"fields": ["doc.*_NORMALISEE"]
}
}
}'

I cross that I will pluck my hair !!

Six Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Hi @mbaroudi

no way with tREST or tRESTClient components. Eventually, I used tSystem component.

 

Regards,

Lorenzo

Ten Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

GET - normally not support body, and it relate not only to Talend (You can try to do this in many programs, like Paw )

 

but You just need change GET to POST, and result would be same:

CURL:

Screen Shot 2017-08-01 at 3.52.15 PM.png

 

Talend:

Screen Shot 2017-08-01 at 3.52.38 PM.png

 

 

Screen Shot 2017-08-01 at 3.54.57 PM.png

 

Screen Shot 2017-08-01 at 3.57.58 PM.png

 

Screen Shot 2017-08-01 at 3.55.07 PM.png

-----------
Four Stars

Re: curl (method: GET) performed in Talend: is possible to pass "-data" parameter with a tREST or tRESTClient component?

Thank you vapukov , it's work know ! Smiley Happy