Skip to main content
Announcements
A fresh, new look for the Data Integration & Quality forums and navigation! Read more about what's changed.
cancel
Showing results for 
Search instead for 
Did you mean: 
msminek
Creator

Talend Open Studio v. 7.1.1 cannot download additional packages

Hi,

 

The  "Install Additional Packages" and the download functionality in the module view didn't work in my Talend Open Studio for Data Integration v7.1.1.
I  found some important files under https://update.talend.com/Studio/tos/7.1.1/ are disappeard. It seems to be a problem.
How can I fix it ? 

 

Regards
minek

 

Labels (2)
1 Solution

Accepted Solutions
msminek
Creator
Author

Hi,

Thank you for your testing. Yes, maybe you are right. There might be a bit tricky network traffic pattern caused by the proxy server, the vaccine solution or so on.

I'm writing the auto retry hack for the tcommon-studio-se project. It's almost done. Code is below. I will go to jila later.

And I also find the timeout setting in the downloding from nexus. NexusServerUtils.getTimeout() tells me that if nothing set in the preference page, the default value is 20000 millisecs. If we have the preference page for the value, this problem will be solved - or still needs retrying.

https://github.com/Talend/tcommon-studio-se/blob/master/main/plugins/org.talend.core.runtime/src/mai...

 

diff --git a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
index 7d023dfd40..58a3298dda 100644
--- a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
+++ b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
@@ -86,6 +86,41 @@ public abstract class HttpClientTransport {
     }
 
     public void doRequest(IProgressMonitor monitor, final URI requestURI) throws Exception {
+        int retries = 5;
+        long waitMillis = 20000;
+
+        int timeout = NexusServerUtils.getTimeout();
+        boolean fTimeout = false;
+
+        for(int t = 1; t <= retries; t++){
+            try{
+                fTimeout = false;
+                doRequestOne(monitor, requestURI);
+            } catch (java.net.SocketTimeoutException e){
+                // Read timed out
+                fTimeout = true;
+                if(t == retries){
+                    throw new Exception(e);
+                }
+            } catch (Exception e) {
+                // DEBUG //
+                System.err.printf("[%d] Exception occured for %s\n", t, requestURI);
+                e.printStackTrace(System.err);
+                // DEBUG //
+                throw e;
+            }
+            if(!fTimeout){
+                break;
+            }
+            // DEBUG //
+            System.err.printf("[%d] Read timeout (in %d millisecs) occured for %s\n", t, timeout, requestURI);
+            // DEBUG //
+
+            Thread.sleep(waitMillis * t);
+        }
+    }
+
+    public void doRequestOne(IProgressMonitor monitor, final URI requestURI) throws Exception {
         if (monitor == null) {
             monitor = new NullProgressMonitor();
         }
@@ -115,6 +150,9 @@ public abstract class HttpClientTransport {
         } catch (org.apache.http.conn.HttpHostConnectException e) {
             // connection failure
             throw e;
+        } catch (java.net.SocketTimeoutException e){
+            // Read timed out
+            throw e;
         } catch (Exception e) {
             throw new Exception(requestURI.toString(), e);
         } finally {
@@ -147,13 +185,15 @@ public abstract class HttpClientTransport {
                         address.getAddress(), proxyPort, "Http Proxy", "Http proxy authentication", null);
                 if (proxyAuthentication != null) {
                     String proxyUser = proxyAuthentication.getUserName();
-                    String proxyPassword = "";
-                    char[] passwordChars = proxyAuthentication.getPassword();
-                    if (passwordChars != null) {
-                        proxyPassword = new String(passwordChars);
+                    if(StringUtils.isNotBlank(proxyUser)){
+                        String proxyPassword = "";
+                        char[] passwordChars = proxyAuthentication.getPassword();
+                        if (passwordChars != null) {
+                            proxyPassword = new String(passwordChars);
+                        }
+                        httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort),
+                                new UsernamePasswordCredentials(proxyUser, proxyPassword));
                     }
-                    httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort),
-                            new UsernamePasswordCredentials(proxyUser, proxyPassword));
                 }
                 HttpHost proxyHost = new HttpHost(proxyServer, proxyPort);
                 httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxyHost);
diff --git a/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java b/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java
index 5f62a9bc6e..f190ce3c2c 100644
--- a/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java
+++ b/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java
@@ -91,6 +91,7 @@ public class NexusDownloader implements IDownloadHelper {
 
             NullProgressMonitor monitor = new NullProgressMonitor();
             ArtifactRepositoryBean nServer = getNexusServer();
+            try{
             new HttpClientTransport(nServer.getRepositoryURL(), nServer.getUserName(), nServer.getPassword()) {
 
                 @Override
@@ -145,9 +146,6 @@ public class NexusDownloader implements IDownloadHelper {
                             if (bos != null) {
                                 bos.close();
                             }
-                            if (tempFolder != null) {
-                                FilesUtils.deleteFile(tempFolder, true);
-                            }
                         }
 
                     }
@@ -155,6 +153,11 @@ public class NexusDownloader implements IDownloadHelper {
                 }
 
             }.doRequest(monitor, parseMvnUrl);
+            }finally{
+                if (tempFolder != null) {
+                    FilesUtils.deleteFile(tempFolder, true);
+                }
+            }
         }
 
     }

 

Thank you for your cooperating.

 

Regards,

minek

View solution in original post

17 Replies
Anonymous
Not applicable

Hello,

What's the error message are you getting when try to downloaf additional packages on Talend Open Studio v. 7.1.1? Is your network OK with you? Are you behind proxy? More information will be preferred.

Best regards

Sabrina

 

 

msminek
Creator
Author

Hi Sabrina,

I have a fresh copy of TOSDI, then I have a bit difference error before.

"Install Additional Packages" function shows the features, now. Then I select "Required third-party libraries(47)" and  click Finish. These functions works fine. So when I push the finish button on the Review Licenses dialog, Error dialog pops up. It said "An error occured (URL) See log for mor details. And I found the logs in my workspace/.metadata/.log file below.

!ENTRY org.talend.platform.logging 4 0 2018-11-13 19:00:03.539
!MESSAGE 2018-11-13 19:00:03,538 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download jersey-client-1.17.jar failed!

!STACK 0
java.lang.Exception: Download jersey-client-1.17.jar failed!
	at org.talend.librariesmanager.utils.DownloadModuleRunnable.downLoad(DownloadModuleRunnable.java:125)
	at org.talend.librariesmanager.utils.DownloadModuleRunnable.run(DownloadModuleRunnable.java:74)
	at org.talend.updates.runtime.model.TalendWebServiceUpdateExtraFeature.install(TalendWebServiceUpdateExtraFeature.java:96)
	at org.talend.updates.runtime.engine.InstallNewFeatureJob.installFeature(InstallNewFeatureJob.java:74)
	at org.talend.updates.runtime.engine.InstallNewFeatureJob.run(InstallNewFeatureJob.java:65)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Caused by: java.lang.Exception: https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/jersey-client-1.17/6.1.0/jersey-client-1.17-6.1.0.jar
	at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:119)
	at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:85)
	at org.talend.librariesmanager.nexus.utils.NexusDownloader.download(NexusDownloader.java:157)
	at org.talend.core.download.DownloadHelperWithProgress.download(DownloadHelperWithProgress.java:90)
	at org.talend.librariesmanager.utils.nexus.NexusDownloadHelperWithProgress.download(NexusDownloadHelperWithProgress.java:95)
	at org.talend.librariesmanager.utils.DownloadModuleRunnable.downLoad(DownloadModuleRunnable.java:110)
	... 5 more
Caused by: java.lang.IllegalArgumentException: Username may not be null
	at org.apache.http.auth.UsernamePasswordCredentials.(UsernamePasswordCredentials.java:81)
	at org.talend.core.nexus.HttpClientTransport.addProxy(HttpClientTransport.java:156)
	at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:111)
	... 10 more

 

> Is your network OK with you? Are you behind proxy?

Today, I download TOS archive from Talend, so I think that's ok for me. But my network behind the proxy with no authorization.

It causes "java.lang.IllegalArgumentException: Username may not be null"? I had seen this error before, but I cannot remember the causes.

 

And Both of "Manual" and "Native" Active Provider in Network Connections page at Preferences raised same error.

Do you have any suggestion for me ?

 

Regards,

minek

msminek
Creator
Author

Hi all,

 

I have a workaround for this issue. It's not a solution, but just a workaround.

  1. Go Talend/studio-se-master github project.
  2. Clone plojects.
  3. Apply the monkey patch below to the tcommon-studio-se  project. It's jut remove the "proxy user is null" problem.
  4. Build your snapshot.
  5. Extract TOS zip file and Run.
  6. Errors are gone.

 

diff --git a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
index 7d023dfd40..2a591efa9b 100644
--- a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
+++ b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
@@ -147,13 +147,15 @@ public abstract class HttpClientTransport {
                         address.getAddress(), proxyPort, "Http Proxy", "Http proxy authentication", null);
                 if (proxyAuthentication != null) {
                     String proxyUser = proxyAuthentication.getUserName();
-                    String proxyPassword = "";
-                    char[] passwordChars = proxyAuthentication.getPassword();
-                    if (passwordChars != null) {
-                        proxyPassword = new String(passwordChars);
+                    if(StringUtils.isNotBlank(proxyUser)){
+                        String proxyPassword = "";
+                        char[] passwordChars = proxyAuthentication.getPassword();
+                        if (passwordChars != null) {
+                            proxyPassword = new String(passwordChars);
+                        }
+                        httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort),
+                                new UsernamePasswordCredentials(proxyUser, proxyPassword));
                     }
-                    httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort),
-                            new UsernamePasswordCredentials(proxyUser, proxyPassword));
                 }
                 HttpHost proxyHost = new HttpHost(proxyServer, proxyPort);
                 httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxyHost);

 

Regards

minek

msminek
Creator
Author

Good  morning all,

 

Finally my download during last nigth is failed by another error.

 

!ENTRY org.talend.platform.logging 4 0 2018-11-14 10:24:55.660
!MESSAGE 2018-11-14 10:24:55,660 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-0.12.0.2.0.6.0-76.jar failed!

!STACK 0
java.lang.Exception: Download hive-exec-0.12.0.2.0.6.0-76.jar failed!
	at org.talend.librariesmanager.utils.DownloadModuleRunnable.downLoad(DownloadModuleRunnable.java:125)
	at org.talend.librariesmanager.utils.DownloadModuleRunnable.run(DownloadModuleRunnable.java:74)
	at org.talend.updates.runtime.model.TalendWebServiceUpdateExtraFeature.install(TalendWebServiceUpdateExtraFeature.java:96)
	at org.talend.updates.runtime.engine.InstallNewFeatureJob.installFeature(InstallNewFeatureJob.java:74)
	at org.talend.updates.runtime.engine.InstallNewFeatureJob.run(InstallNewFeatureJob.java:65)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Caused by: java.lang.Exception: https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec-0.12.0.2.0.6.0-76/6.0.0/hive-exec-0.12.0.2.0.6.0-76-6.0.0.jar
	at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:119)
	at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:85)
	at org.talend.librariesmanager.nexus.utils.NexusDownloader.download(NexusDownloader.java:157)
	at org.talend.core.download.DownloadHelperWithProgress.download(DownloadHelperWithProgress.java:90)
	at org.talend.librariesmanager.utils.nexus.NexusDownloadHelperWithProgress.download(NexusDownloadHelperWithProgress.java:95)
	at org.talend.librariesmanager.utils.DownloadModuleRunnable.downLoad(DownloadModuleRunnable.java:110)
	... 5 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at org.talend.librariesmanager.nexus.utils.NexusDownloader$1.execute(NexusDownloader.java:100)
	at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:112)
	... 10 more

 

I'm still investigating. Currently I dought insuficcent timeout had set at HttpClientTransport#doRequest().

Any suggestions are welcome.

 

Regards

minek

Anonymous
Not applicable

Hello,

In Studio,  please navigate to Window > Preferences > Talend > Performance to increase timeout value to see if it is OK with you.

Best regards

Sabrina

msminek
Creator
Author

Hi,

Thanks to reply.

 

0683p000009M11d.png

 

I need some clarification. Did you mean "timeout value" is this pages "Connection timeout (seconds)" that is activated by to check "Activate the timeout for database connection." ? If so, the setting seems to be nothing to do for this case when I tried.

 

 

 

 

 

 

Regards

minek

Anonymous
Not applicable

Hello,

Does this issue happen all the time or only happens very often when download all Optional third-party libraries at one time?

The default socket read timeout is 60s, there is one way is to increase the readtimeout parameters in the studio ini file

-Dsun.net.client.defaultReadTimeout=600000

 Is it OK with you?

Best regards

Sabrina

 

msminek
Creator
Author

Hi,

> Does this issue happen all the time or only happens very often when download all Optional third-party libraries at one time?

I'd tested "Install Additional Packages" several times I heard that.

  1. First time,  Optional modules is 1135. then ended by timeout.
  2. 1134, timeout
  3. 1132, timeout
  4. 1130, timeout
  5. 1129, timeout

Tommorow, I will test with an fresh copy of 7.1.1.

 

-Dsun.net.client.defaultReadTimeout=600000

Does it mean 600,000 millsecs = 600 secs = 10mins ?

I just copy and paste the line in my studio.ini, but "timeout" errors didn't take a time I saw.

 

Regard

minek

 

msminek
Creator
Author

Hi,

 I had a little time today. so I focused on "install Additional Packages" function with a fresh copy. And it's not finished yet, but

Required modules - no error.
Optional modules - errors at

Errors at similar modules yesterday. But I haven't check those are exact same.

 

May be I don't need understand this phenomenon, But I need the "retry download on failure" functionality.

 

Regards

minek