topic Re: [resolved] Retrieve Xml file from an Url in Talend Studio

[resolved] Retrieve Xml file from an Url

Anonymous — Mon, 26 Aug 2013 14:55:36 GMT

Hello,
I would like to save an xml file retrive from an Url.
In order to do that, I'm using a tFileFetch component but I have the following error :
Exception in component tFileFetch_1
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at portefeuille_eg.test_0_1.test$1SocketFactory_tFileFetch_1.createSocket(test.java:382)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:706)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
I've checked in the advance setting "use authentication".
And if I call the url in internet explorer, a windows authentification windows open and after writing my ID, the xml appears well in internet explorer.
Someone have an idea please ?
Regards

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Tue, 27 Aug 2013 03:23:53 GMT

Hi,
You can have a try to change the User Authentication:IE-->Tool-->Security-->Local Intranet Zone-->Custom level-->User Authentication log on -->Automatic logon with current user name and password to see if this windows authentification windows open?
Best regards
Sabrina

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Tue, 27 Aug 2013 09:19:07 GMT

Hi Sabrina,
Thanks you for your answer.
I try it but I still have the same error.
Regards,
Stef

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Tue, 27 Aug 2013 10:25:40 GMT

Hi,
I suspect it is caused by your firewall. Do you still have this "windows authentification windows open" after close your firewall?
Best regards
Sabrina

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Tue, 27 Aug 2013 10:46:40 GMT

Sorry Sabrina, I'm not sure if we understand well...
I have this windows authentification when I call the url with internet explorer.
I do that in order to test my url but what I need is to call the url with my talend job. And in order to provide this authentication, I've thinked that using advance setting provide me this problem.
But It's not the case ...
Regards,
Stef

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Tue, 27 Aug 2013 10:58:21 GMT

Sabrina, I think I've understood what you would like to say.
I have to set the parameters of my proxy in Talend.
I will take a look and let you know...
Regards,
Stef

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Tue, 27 Aug 2013 14:01:38 GMT

By adding the proxy parameters, I still have a problem :
7 août 2013 14:58:40 org.apache.commons.httpclient.auth.AuthChallengeProcessor selectAuthScheme
INFO: basic authentication scheme selected
27 août 2013 14:58:40 org.apache.commons.httpclient.HttpMethodBase readResponseBody
INFO: Response content length is not known
27 août 2013 14:58:41 org.apache.commons.httpclient.auth.AuthChallengeProcessor selectAuthScheme
INFO: basic authentication scheme selected
27 août 2013 14:58:41 org.apache.commons.httpclient.HttpMethodDirector processAuthenticationResponse
GRAVE: Invalid challenge: Basic
org.apache.commons.httpclient.auth.MalformedChallengeException: Invalid challenge: Basic
at org.apache.commons.httpclient.auth.AuthChallengeParser.extractParams(AuthChallengeParser.java:97)
at org.apache.commons.httpclient.auth.RFC2617Scheme.processChallenge(RFC2617Scheme.java:93)
at org.apache.commons.httpclient.auth.BasicScheme.processChallenge(BasicScheme.java:111)
at org.apache.commons.httpclient.auth.AuthChallengeProcessor.processChallenge(AuthChallengeProcessor.java:161)
at org.apache.commons.httpclient.HttpMethodDirector.processWWWAuthChallenge(HttpMethodDirector.java:689)
at org.apache.commons.httpclient.HttpMethodDirector.processAuthenticationResponse(HttpMethodDirector.java:663)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:192)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at integration_conso_dwh.test_0_1.test.tFileFetch_1Process(test.java:593)
at integration_conso_dwh.test_0_1.test$1.run(test.java:970)
Exception in component tFileFetch_1
java.lang.Exception: Method failed: HTTP/1.1 401 Unauthorized
at integration_conso_dwh.test_0_1.test.tFileFetch_1Process(test.java:596)
at integration_conso_dwh.test_0_1.test$1.run(test.java:970)
828 milliseconds

Any ideas please ?
Regards,
Stef

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Thu, 29 Aug 2013 10:55:07 GMT

Hello Sabrina,
After some tests with the system administrator, it's not a problem of firewall or proxy.
The connection to the https seems to be ok but I can't save the answer ...
Regards,
Stef

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Thu, 29 Aug 2013 11:14:36 GMT

Hi,

I can't save the answer ...

Do you mean you cannot store the retrieved file into destination directory? Could you please give us more info about your current issue?
Best regards
Sabrina

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Fri, 30 Aug 2013 08:16:58 GMT

Hi,
Yes I can't store the retrieved file into destination directory.
I tried the save cookie but I have the same issue.
For my system administrator, the connection to the https seems to be done ... So I don't understand.

Regards,
Stef

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Fri, 30 Aug 2013 08:26:04 GMT

Hi,
Did you check out "Die on error" option on tFileList Component to see if there is any error. Is there any response from server in console?
Best regards
Sabrina

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Fri, 30 Aug 2013 08:41:08 GMT

Yes, these option is check and it return the same error.
connecting to socket on port 3547
30 août 2013 09:36:50 org.apache.commons.httpclient.auth.AuthChallengeProcessor selectAuthScheme
INFO: basic authentication scheme selected
connected
30 août 2013 09:36:50 org.apache.commons.httpclient.HttpMethodBase readResponseBody
INFO: Response content length is not known
30 août 2013 09:36:51 org.apache.commons.httpclient.auth.AuthChallengeProcessor selectAuthScheme
INFO: basic authentication scheme selected
30 août 2013 09:36:51 org.apache.commons.httpclient.HttpMethodDirector processAuthenticationResponse
GRAVE: Invalid challenge: Basic
org.apache.commons.httpclient.auth.MalformedChallengeException: Invalid challenge: Basic
at org.apache.commons.httpclient.auth.AuthChallengeParser.extractParams(AuthChallengeParser.java:97)
at org.apache.commons.httpclient.auth.RFC2617Scheme.processChallenge(RFC2617Scheme.java:93)
at org.apache.commons.httpclient.auth.BasicScheme.processChallenge(BasicScheme.java:111)
at org.apache.commons.httpclient.auth.AuthChallengeProcessor.processChallenge(AuthChallengeProcessor.java:161)
at org.apache.commons.httpclient.HttpMethodDirector.processWWWAuthChallenge(HttpMethodDirector.java:689)
at org.apache.commons.httpclient.HttpMethodDirector.processAuthenticationResponse(HttpMethodDirector.java:663)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:192)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at integration_conso_dwh.test_0_1.test.tFileFetch_1Process(test.java:668)
at integration_conso_dwh.test_0_1.test$1.run(test.java:1085)
Exception in component tFileFetch_1
java.lang.Exception: Method failed: HTTP/1.1 401 Unauthorized
at integration_conso_dwh.test_0_1.test.tFileFetch_1Process(test.java:671)
at integration_conso_dwh.test_0_1.test$1.run(test.java:1085)
disconnected
Job test terminé à 09:36 30/08/2013.

Regards,
Stef

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Fri, 30 Aug 2013 08:47:59 GMT

Hi,
There is a response given by shong in https://community.talend.com/t5/Design-and-Development/problem-downloading-file-from-https/td-p/91239
Would you mind checking it?
Best regards
Sabrina

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Tue, 03 Sep 2013 09:59:44 GMT

Hello Sabrina,
I've seen with people in charge of this url, and they have noticed that the user/password is pass in the header http but also in the URL.
For them, there is a conflict and I should just pass the authentification by the header http. There is something in the tFileFecth which add automatically the authentifcation in the URL.
Do you know what ?

Regards,
Stephane

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Thu, 05 Sep 2013 10:13:07 GMT

Hi,

I've seen with people in charge of this url, and they have noticed that the user/password is pass in the header http but also in the URL.
For them, there is a conflict and I should just pass the authentification by the header http. There is something in the tFileFecth which add automatically the authentifcation in the URL.

There are four protocols: http, https,ftp and smb. It seems that there is no such option adding automatically the authentifcation in the URL in the tFileFecth.
You can open a jira issue on talend bug tracker to see if there is a solution for that.
Best regards
Sabrina

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Mon, 09 Sep 2013 14:28:22 GMT

Hello Sabrina,
Finally, peoples in charge of the url can only see my call when I'm using internet explorer.
The conflict of authentification is not done by the tFileFech ...
But something in the tfilefech generates a HTTP/1.1 401 Unauthorized exception.
Proxy parameters are good.
Maybe the format of my url pose problem to the tFileFech component ?
thsi is my url :
https://xxxxxxxx.com/v1/xml/GetData?&start=2013-07-16&end=2013-07-16&max-results=30&columns=lortal_site,lage,m:visits&sort=-m:visits&site=380240&start-index=1
I've tried to call the url https://xxxxxxxx.com/v1/xml/GetData by passing parameters (start, end, max-results, columns, sort, site, start-index) with the post method but I have the same problem.
Regards,
Stephane

Re: [resolved] Retrieve Xml file from an Url

Anonymous — Mon, 07 Oct 2013 15:24:23 GMT

Finally, I've solved my problem by :
- make an exception in the firewall for my url
- using a tHttpRequest with authentication
Regards,
Stephane