One Star

[resolved] POSTing a file to a website

Hi everyone!
I've got a task which I'd love to use TOS for but I can't for the life of me work out how to accomplish it.
There is a website (protected by apache basic auth) which requires me to POST a file (which happens to be a CSV). I can manage the first bit - getting the data, but I'm not sure about the second and third bits, namely forming this data into a CSV file and then using HTTP POST to send it to this other website.
For those that know Perl, here is a snippet of the code we have which is doing what I'd like TOS to do:
         
my $ua = BrowserUA->new(
agent => "Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
);
my $req = POST $url,
"Content-Type" => 'multipart/form-data',
"Content" => ,
];

Has anyone got any hints that they can give me? I'm especially confused about the final bit of POSTing to the website as I can't see how to post a file (values aren't IN a file, I want to upload the actual file). I believe the content type above is making this happen, but I can't see how to do this in TOS.
Thanks in advance.
1 ACCEPTED SOLUTION

Accepted Solutions
One Star

Re: [resolved] POSTing a file to a website

I've solved this!!
The key was not to use tHttpRequest at all. The correct component to use is tFileFetch. Despite it's name, it can send a file too. Worked straight away, very simple.
14 REPLIES
Six Stars

Re: [resolved] POSTing a file to a website

Hey,
first of all, what's your source data format? When these data will be managed to CSV you could use tHttpRequest component which is able to do the POST file over HTTP protocol with custom headers if needed and then write response of the server to file, so you are able to check status of finished POST. This could be really easy, post some source data.
Ladislav
One Star

Re: [resolved] POSTing a file to a website

Hi,
Thanks for the reply.
For the proof of concept I'm doing, I'm short-cutting the generation of the file for now and I'm focussing only on POSTing the file to the receiving website. To do this, I've just hand-created a CSV file.
I have tried using tHttpRequest - even as the sole component in a job with everything hardcoded. In other words, I had for the Headers section:
"Content-Type" "multipart/form-data"
"submit" "1"
"upfile" "C:\XXXXXX\YYYY.csv"
I also set:
Method: POST
And the appropriate authentication.
The job seems to run. It pauses after reporting "connected" (which I assume is my 104 line CSV file being uploaded) and some seconds later reports "disconnected" and exits with a code 0. However, the other system doesn't appear to have picked anything up.
That's currently where I am right now. If you have any advice for me on debugging this (or whether the way I've gone about this is wrong), then please let me know.
Thanks.
One Star

Re: [resolved] POSTing a file to a website

Oh, I forgot to mention that I've tried putting the server response back to a file and the file is 0 bytes....
One Star

Re: [resolved] POSTing a file to a website

A picture is worth 1000 words....
Six Stars

Re: [resolved] POSTing a file to a website

Well,
this component comes from my request in the past because I needed it also for my project, take a look at this component tFileHttpPost at https://exchange.talend.com/#marketplacesearch:gallery=marketplace%252F1&ob=releaseDate&o=0&c=20&d=t...
I do not use custom headers in my scenario except one which indicates that I post text or XML (text/xml) content as you can see at attached picture. Try to change your configuration in a way showed on picture, use "Post parameters from file", this is the attribute which point to file which schould be posted over http.
Ladislav

Six Stars

Re: [resolved] POSTing a file to a website

I also recommend use wireshark or other network capturing tool. Wireshark support function Follow TCP Stream, so once you find the packet related to your request, using this function wireshark show you whole request response, so you see what was sent and received.
Ladislav
Six Stars

Re: [resolved] POSTing a file to a website

If nothing from what proposed works, you could write your own subroutine in talend like this:
URL dest = "http://www.mysite.com "; 
URLConnection urlCon = dest.openConnection();
// prepare input and output
urlCon.setDoInput(true);
urlCon.setDoOutput(true);
// Disable caching
urlCon.setUseCaches(false);
// Post output
DataOutputStream out =
new DataOutputStream(urlCon.getOutputStream());
out.writeBytes( file.toString );
out.flush();
out.close();
// return servlet response as a DataInputStream
DataInputStream in = new DataInputStream(urlCon.getInputStream());
// and so on
....

There is also HTTP Client package provided by Apache Jakarta at http://hc.apache.org/downloads.cgi which make it easier to work with http.
Ladislav
One Star

Re: [resolved] POSTing a file to a website

Thanks for your info. I have tried re-doing my job to match more closely like yours but it's exactly the same situation.
I think I need to try WireShark to see what is going on as I get no feedback at all from TOS or the other website. I'd rather not have to write specific code for this, otherwise I will just stick with my Perl script Smiley Happy
Six Stars

Re: [resolved] POSTing a file to a website

Wireshark will show the pure traffic, but you could also try to debug Talend session. When starting job from Talend you could use Debug trace or pure debug session to see what's go'in on. Let me know the results...
Ladislav
One Star

Re: [resolved] POSTing a file to a website

I've solved this!!
The key was not to use tHttpRequest at all. The correct component to use is tFileFetch. Despite it's name, it can send a file too. Worked straight away, very simple.
Six Stars

Re: [resolved] POSTing a file to a website

Nice :-) But still I don't know why the tHttpRequest didn't work for you.
One Star

Re: [resolved] POSTing a file to a website

Yep, that is a mystery, and if I needed to POST the file as well as other things, I may well have been stuck. I might look into it one day if I get the time. For now though, I can get on with my project Smiley Happy
Six Stars

Re: [resolved] POSTing a file to a website

Hi aiclara,
I need to to do the same thing as you did. Can you show me how to use tFileFetch instead of tHttpRequest?
Thanks.
One Star

Re: [resolved] POSTing a file to a website

You guysssssss, come on since when can you make a http request to a https URL. Haha lol your missing a whole  layer of encryption. If no one has figured it out after a difference of more than 3 years! Cheers.