[resolved] POSTing a file to a website

[Anonymous]

Hi everyone!
I've got a task which I'd love to use TOS for but I can't for the life of me work out how to accomplish it.
There is a website (protected by apache basic auth) which requires me to POST a file (which happens to be a CSV). I can manage the first bit - getting the data, but I'm not sure about the second and third bits, namely forming this data into a CSV file and then using HTTP POST to send it to this other website.
For those that know Perl, here is a snippet of the code we have which is doing what I'd like TOS to do:

         
         my $ua = BrowserUA->new(
                 agent => "Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
         );
         my $req = POST $url,
                 "Content-Type" => 'multipart/form-data',
                 "Content" => ,
                 ];

Has anyone got any hints that they can give me? I'm especially confused about the final bit of POSTing to the website as I can't see how to post a file (values aren't IN a file, I want to upload the actual file). I believe the content type above is making this happen, but I can't see how to do this in TOS.
Thanks in advance.

[Anonymous]

Hey,
first of all, what's your source data format? When these data will be managed to CSV you could use tHttpRequest component which is able to do the POST file over HTTP protocol with custom headers if needed and then write response of the server to file, so you are able to check status of finished POST. This could be really easy, post some source data.
Ladislav

[Anonymous]

Hi,
Thanks for the reply.
For the proof of concept I'm doing, I'm short-cutting the generation of the file for now and I'm focussing only on POSTing the file to the receiving website. To do this, I've just hand-created a CSV file.
I have tried using tHttpRequest - even as the sole component in a job with everything hardcoded. In other words, I had for the Headers section:
"Content-Type" "multipart/form-data"
"submit" "1"
"upfile" "C:\XXXXXX\YYYY.csv"
I also set:
Method: POST
And the appropriate authentication.
The job seems to run. It pauses after reporting "connected" (which I assume is my 104 line CSV file being uploaded) and some seconds later reports "disconnected" and exits with a code 0. However, the other system doesn't appear to have picked anything up.
That's currently where I am right now. If you have any advice for me on debugging this (or whether the way I've gone about this is wrong), then please let me know.
Thanks.

[Anonymous]

Oh, I forgot to mention that I've tried putting the server response back to a file and the file is 0 bytes....

[Anonymous]

A picture is worth 1000 words....

[Anonymous]

Well,
this component comes from my request in the past because I needed it also for my project, take a look at this component tFileHttpPost at https://exchange.talend.com/#marketplacesearch:gallery=marketplace%252F1&ob=releaseDate&o=0&c=20&d=t...
I do not use custom headers in my scenario except one which indicates that I post text or XML (text/xml) content as you can see at attached picture. Try to change your configuration in a way showed on picture, use "Post parameters from file", this is the attribute which point to file which schould be posted over http.
Ladislav

[Anonymous]

I also recommend use wireshark or other network capturing tool. Wireshark support function Follow TCP Stream, so once you find the packet related to your request, using this function wireshark show you whole request response, so you see what was sent and received.
Ladislav

[Anonymous]

If nothing from what proposed works, you could write your own subroutine in talend like this:

URL dest = "http://www.mysite.com "; 
URLConnection urlCon = dest.openConnection();
// prepare input and output 
urlCon.setDoInput(true); 
urlCon.setDoOutput(true);
// Disable caching 
urlCon.setUseCaches(false);
// Post output 
DataOutputStream out = 
new DataOutputStream(urlCon.getOutputStream()); 
out.writeBytes( file.toString ); 
out.flush(); 
out.close();
// return servlet response as a DataInputStream 
DataInputStream in = new DataInputStream(urlCon.getInputStream());
// and so on
....

There is also HTTP Client package provided by Apache Jakarta at http://hc.apache.org/downloads.cgi which make it easier to work with http.
Ladislav

[Anonymous]

Thanks for your info. I have tried re-doing my job to match more closely like yours but it's exactly the same situation.
I think I need to try WireShark to see what is going on as I get no feedback at all from TOS or the other website. I'd rather not have to write specific code for this, otherwise I will just stick with my Perl script

[Anonymous]

Wireshark will show the pure traffic, but you could also try to debug Talend session. When starting job from Talend you could use Debug trace or pure debug session to see what's go'in on. Let me know the results...
Ladislav