topic Re: [resolved] POSTing a file to a website in Talend Studio

[resolved] POSTing a file to a website

Anonymous — Sat, 16 Nov 2024 12:34:14 GMT

Hi everyone!
I've got a task which I'd love to use TOS for but I can't for the life of me work out how to accomplish it.
There is a website (protected by apache basic auth) which requires me to POST a file (which happens to be a CSV). I can manage the first bit - getting the data, but I'm not sure about the second and third bits, namely forming this data into a CSV file and then using HTTP POST to send it to this other website.
For those that know Perl, here is a snippet of the code we have which is doing what I'd like TOS to do:

         
         my $ua = BrowserUA->new(
                 agent => "Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
         );
         my $req = POST $url,
                 "Content-Type" => 'multipart/form-data',
                 "Content" => ,
                 ];

Has anyone got any hints that they can give me? I'm especially confused about the final bit of POSTing to the website as I can't see how to post a file (values aren't IN a file, I want to upload the actual file). I believe the content type above is making this happen, but I can't see how to do this in TOS.
Thanks in advance.

Re: [resolved] POSTing a file to a website

Anonymous — Wed, 30 Nov 2011 21:11:29 GMT

Hey,
first of all, what's your source data format? When these data will be managed to CSV you could use tHttpRequest component which is able to do the POST file over HTTP protocol with custom headers if needed and then write response of the server to file, so you are able to check status of finished POST. This could be really easy, post some source data.
Ladislav

Re: [resolved] POSTing a file to a website

Anonymous — Thu, 01 Dec 2011 08:47:11 GMT

Hi,
Thanks for the reply.
For the proof of concept I'm doing, I'm short-cutting the generation of the file for now and I'm focussing only on POSTing the file to the receiving website. To do this, I've just hand-created a CSV file.
I have tried using tHttpRequest - even as the sole component in a job with everything hardcoded. In other words, I had for the Headers section:
"Content-Type" "multipart/form-data"
"submit" "1"
"upfile" "C:\XXXXXX\YYYY.csv"
I also set:
Method: POST
And the appropriate authentication.
The job seems to run. It pauses after reporting "connected" (which I assume is my 104 line CSV file being uploaded) and some seconds later reports "disconnected" and exits with a code 0. However, the other system doesn't appear to have picked anything up.
That's currently where I am right now. If you have any advice for me on debugging this (or whether the way I've gone about this is wrong), then please let me know.
Thanks.

Re: [resolved] POSTing a file to a website

Anonymous — Thu, 01 Dec 2011 08:54:04 GMT

Oh, I forgot to mention that I've tried putting the server response back to a file and the file is 0 bytes....

Re: [resolved] POSTing a file to a website

Anonymous — Thu, 01 Dec 2011 09:08:25 GMT

A picture is worth 1000 words....

Re: [resolved] POSTing a file to a website

Anonymous — Thu, 01 Dec 2011 09:34:02 GMT

Well,
this component comes from my request in the past because I needed it also for my project, take a look at this component tFileHttpPost at https://exchange.talend.com/#marketplacesearch:gallery=marketplace%252F1&ob=releaseDate&o=0&c=20&d=true&s=tFileHttpPost%2520&f=&tf=&bf=&cat=&title=Search%2520Results
I do not use custom headers in my scenario except one which indicates that I post text or XML (text/xml) content as you can see at attached picture. Try to change your configuration in a way showed on picture, use "Post parameters from file", this is the attribute which point to file which schould be posted over http.
Ladislav

Re: [resolved] POSTing a file to a website

Anonymous — Thu, 01 Dec 2011 09:53:39 GMT

I also recommend use wireshark or other network capturing tool. Wireshark support function Follow TCP Stream, so once you find the packet related to your request, using this function wireshark show you whole request response, so you see what was sent and received.
Ladislav

Re: [resolved] POSTing a file to a website

Anonymous — Thu, 01 Dec 2011 09:57:26 GMT

If nothing from what proposed works, you could write your own subroutine in talend like this:

URL dest = "http://www.mysite.com "; 
URLConnection urlCon = dest.openConnection();
// prepare input and output 
urlCon.setDoInput(true); 
urlCon.setDoOutput(true);
// Disable caching 
urlCon.setUseCaches(false);
// Post output 
DataOutputStream out = 
new DataOutputStream(urlCon.getOutputStream()); 
out.writeBytes( file.toString ); 
out.flush(); 
out.close();
// return servlet response as a DataInputStream 
DataInputStream in = new DataInputStream(urlCon.getInputStream());
// and so on
....

There is also HTTP Client package provided by Apache Jakarta at http://hc.apache.org/downloads.cgi which make it easier to work with http.
Ladislav

Re: [resolved] POSTing a file to a website

Anonymous — Thu, 01 Dec 2011 10:05:52 GMT

Thanks for your info. I have tried re-doing my job to match more closely like yours but it's exactly the same situation.
I think I need to try WireShark to see what is going on as I get no feedback at all from TOS or the other website. I'd rather not have to write specific code for this, otherwise I will just stick with my Perl script

Re: [resolved] POSTing a file to a website

Anonymous — Thu, 01 Dec 2011 10:10:06 GMT

Wireshark will show the pure traffic, but you could also try to debug Talend session. When starting job from Talend you could use Debug trace or pure debug session to see what's go'in on. Let me know the results...
Ladislav

Re: [resolved] POSTing a file to a website

Anonymous — Mon, 05 Dec 2011 09:01:14 GMT

I've solved this!!
The key was not to use tHttpRequest at all. The correct component to use is tFileFetch. Despite it's name, it can send a file too. Worked straight away, very simple.

Re: [resolved] POSTing a file to a website

Anonymous — Mon, 05 Dec 2011 09:08:50 GMT

Nice 🙂 But still I don't know why the tHttpRequest didn't work for you.

Re: [resolved] POSTing a file to a website

Anonymous — Mon, 05 Dec 2011 09:14:48 GMT

Yep, that is a mystery, and if I needed to POST the file as well as other things, I may well have been stuck. I might look into it one day if I get the time. For now though, I can get on with my project

Re: [resolved] POSTing a file to a website

Anonymous — Fri, 22 Feb 2013 23:02:15 GMT

Hi aiclara,
I need to to do the same thing as you did. Can you show me how to use tFileFetch instead of tHttpRequest?
Thanks.

Re: [resolved] POSTing a file to a website

Anonymous — Wed, 29 Jun 2016 01:24:30 GMT

You guysssssss, come on since when can you make a http request to a https URL. Haha lol your missing a whole layer of encryption. If no one has figured it out after a difference of more than 3 years! Cheers.

Re: [resolved] POSTing a file to a website

joe86 — Mon, 18 Mar 2019 05:07:35 GMT

@aiclara
Hey, this is a pretty old post. By any chance, do have the details of how to use tFileFetch to Upload/Post a file (from my local hard disk) to a web ( https://clientconn.sharefile.com/share/upload/r773a1b7922e44cca)