
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Talend Open Studio v. 7.1.1 cannot download additional packages
Hi,
The "Install Additional Packages" and the download functionality in the module view didn't work in my Talend Open Studio for Data Integration v7.1.1.
I found some important files under https://update.talend.com/Studio/tos/7.1.1/ are disappeard. It seems to be a problem.
How can I fix it ?
Regards
minek
- « Previous Replies
-
- 1
- 2
- Next Replies »
Accepted Solutions

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Thank you for your testing. Yes, maybe you are right. There might be a bit tricky network traffic pattern caused by the proxy server, the vaccine solution or so on.
I'm writing the auto retry hack for the tcommon-studio-se project. It's almost done. Code is below. I will go to jila later.
And I also find the timeout setting in the downloding from nexus. NexusServerUtils.getTimeout() tells me that if nothing set in the preference page, the default value is 20000 millisecs. If we have the preference page for the value, this problem will be solved - or still needs retrying.
diff --git a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java index 7d023dfd40..58a3298dda 100644 --- a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java +++ b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java @@ -86,6 +86,41 @@ public abstract class HttpClientTransport { } public void doRequest(IProgressMonitor monitor, final URI requestURI) throws Exception { + int retries = 5; + long waitMillis = 20000; + + int timeout = NexusServerUtils.getTimeout(); + boolean fTimeout = false; + + for(int t = 1; t <= retries; t++){ + try{ + fTimeout = false; + doRequestOne(monitor, requestURI); + } catch (java.net.SocketTimeoutException e){ + // Read timed out + fTimeout = true; + if(t == retries){ + throw new Exception(e); + } + } catch (Exception e) { + // DEBUG // + System.err.printf("[%d] Exception occured for %s\n", t, requestURI); + e.printStackTrace(System.err); + // DEBUG // + throw e; + } + if(!fTimeout){ + break; + } + // DEBUG // + System.err.printf("[%d] Read timeout (in %d millisecs) occured for %s\n", t, timeout, requestURI); + // DEBUG // + + Thread.sleep(waitMillis * t); + } + } + + public void doRequestOne(IProgressMonitor monitor, final URI requestURI) throws Exception { if (monitor == null) { monitor = new NullProgressMonitor(); } @@ -115,6 +150,9 @@ public abstract class HttpClientTransport { } catch (org.apache.http.conn.HttpHostConnectException e) { // connection failure throw e; + } catch (java.net.SocketTimeoutException e){ + // Read timed out + throw e; } catch (Exception e) { throw new Exception(requestURI.toString(), e); } finally { @@ -147,13 +185,15 @@ public abstract class HttpClientTransport { address.getAddress(), proxyPort, "Http Proxy", "Http proxy authentication", null); if (proxyAuthentication != null) { String proxyUser = proxyAuthentication.getUserName(); - String proxyPassword = ""; - char[] passwordChars = proxyAuthentication.getPassword(); - if (passwordChars != null) { - proxyPassword = new String(passwordChars); + if(StringUtils.isNotBlank(proxyUser)){ + String proxyPassword = ""; + char[] passwordChars = proxyAuthentication.getPassword(); + if (passwordChars != null) { + proxyPassword = new String(passwordChars); + } + httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort), + new UsernamePasswordCredentials(proxyUser, proxyPassword)); } - httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort), - new UsernamePasswordCredentials(proxyUser, proxyPassword)); } HttpHost proxyHost = new HttpHost(proxyServer, proxyPort); httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxyHost); diff --git a/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java b/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java index 5f62a9bc6e..f190ce3c2c 100644 --- a/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java +++ b/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java @@ -91,6 +91,7 @@ public class NexusDownloader implements IDownloadHelper { NullProgressMonitor monitor = new NullProgressMonitor(); ArtifactRepositoryBean nServer = getNexusServer(); + try{ new HttpClientTransport(nServer.getRepositoryURL(), nServer.getUserName(), nServer.getPassword()) { @Override @@ -145,9 +146,6 @@ public class NexusDownloader implements IDownloadHelper { if (bos != null) { bos.close(); } - if (tempFolder != null) { - FilesUtils.deleteFile(tempFolder, true); - } } } @@ -155,6 +153,11 @@ public class NexusDownloader implements IDownloadHelper { } }.doRequest(monitor, parseMvnUrl); + }finally{ + if (tempFolder != null) { + FilesUtils.deleteFile(tempFolder, true); + } + } } }
Thank you for your cooperating.
Regards,
minek

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
What's the error message are you getting when try to downloaf additional packages on Talend Open Studio v. 7.1.1? Is your network OK with you? Are you behind proxy? More information will be preferred.
Best regards
Sabrina

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Sabrina,
I have a fresh copy of TOSDI, then I have a bit difference error before.
"Install Additional Packages" function shows the features, now. Then I select "Required third-party libraries(47)" and click Finish. These functions works fine. So when I push the finish button on the Review Licenses dialog, Error dialog pops up. It said "An error occured (URL) See log for mor details. And I found the logs in my workspace/.metadata/.log file below.
!ENTRY org.talend.platform.logging 4 0 2018-11-13 19:00:03.539 !MESSAGE 2018-11-13 19:00:03,538 ERROR org.talend.commons.exception.CommonExceptionHandler - Download jersey-client-1.17.jar failed! !STACK 0 java.lang.Exception: Download jersey-client-1.17.jar failed! at org.talend.librariesmanager.utils.DownloadModuleRunnable.downLoad(DownloadModuleRunnable.java:125) at org.talend.librariesmanager.utils.DownloadModuleRunnable.run(DownloadModuleRunnable.java:74) at org.talend.updates.runtime.model.TalendWebServiceUpdateExtraFeature.install(TalendWebServiceUpdateExtraFeature.java:96) at org.talend.updates.runtime.engine.InstallNewFeatureJob.installFeature(InstallNewFeatureJob.java:74) at org.talend.updates.runtime.engine.InstallNewFeatureJob.run(InstallNewFeatureJob.java:65) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) Caused by: java.lang.Exception: https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/jersey-client-1.17/6.1.0/jersey-client-1.17-6.1.0.jar at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:119) at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:85) at org.talend.librariesmanager.nexus.utils.NexusDownloader.download(NexusDownloader.java:157) at org.talend.core.download.DownloadHelperWithProgress.download(DownloadHelperWithProgress.java:90) at org.talend.librariesmanager.utils.nexus.NexusDownloadHelperWithProgress.download(NexusDownloadHelperWithProgress.java:95) at org.talend.librariesmanager.utils.DownloadModuleRunnable.downLoad(DownloadModuleRunnable.java:110) ... 5 more Caused by: java.lang.IllegalArgumentException: Username may not be null at org.apache.http.auth.UsernamePasswordCredentials.(UsernamePasswordCredentials.java:81) at org.talend.core.nexus.HttpClientTransport.addProxy(HttpClientTransport.java:156) at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:111) ... 10 more
> Is your network OK with you? Are you behind proxy?
Today, I download TOS archive from Talend, so I think that's ok for me. But my network behind the proxy with no authorization.
It causes "java.lang.IllegalArgumentException: Username may not be null"? I had seen this error before, but I cannot remember the causes.
And Both of "Manual" and "Native" Active Provider in Network Connections page at Preferences raised same error.
Do you have any suggestion for me ?
Regards,
minek

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi all,
I have a workaround for this issue. It's not a solution, but just a workaround.
- Go Talend/studio-se-master github project.
- Clone plojects.
- Apply the monkey patch below to the tcommon-studio-se project. It's jut remove the "proxy user is null" problem.
- Build your snapshot.
- Extract TOS zip file and Run.
- Errors are gone.
diff --git a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java index 7d023dfd40..2a591efa9b 100644 --- a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java +++ b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java @@ -147,13 +147,15 @@ public abstract class HttpClientTransport { address.getAddress(), proxyPort, "Http Proxy", "Http proxy authentication", null); if (proxyAuthentication != null) { String proxyUser = proxyAuthentication.getUserName(); - String proxyPassword = ""; - char[] passwordChars = proxyAuthentication.getPassword(); - if (passwordChars != null) { - proxyPassword = new String(passwordChars); + if(StringUtils.isNotBlank(proxyUser)){ + String proxyPassword = ""; + char[] passwordChars = proxyAuthentication.getPassword(); + if (passwordChars != null) { + proxyPassword = new String(passwordChars); + } + httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort), + new UsernamePasswordCredentials(proxyUser, proxyPassword)); } - httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort), - new UsernamePasswordCredentials(proxyUser, proxyPassword)); } HttpHost proxyHost = new HttpHost(proxyServer, proxyPort); httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxyHost);
Regards
minek

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Good morning all,
Finally my download during last nigth is failed by another error.
!ENTRY org.talend.platform.logging 4 0 2018-11-14 10:24:55.660 !MESSAGE 2018-11-14 10:24:55,660 ERROR org.talend.commons.exception.CommonExceptionHandler - Download hive-exec-0.12.0.2.0.6.0-76.jar failed! !STACK 0 java.lang.Exception: Download hive-exec-0.12.0.2.0.6.0-76.jar failed! at org.talend.librariesmanager.utils.DownloadModuleRunnable.downLoad(DownloadModuleRunnable.java:125) at org.talend.librariesmanager.utils.DownloadModuleRunnable.run(DownloadModuleRunnable.java:74) at org.talend.updates.runtime.model.TalendWebServiceUpdateExtraFeature.install(TalendWebServiceUpdateExtraFeature.java:96) at org.talend.updates.runtime.engine.InstallNewFeatureJob.installFeature(InstallNewFeatureJob.java:74) at org.talend.updates.runtime.engine.InstallNewFeatureJob.run(InstallNewFeatureJob.java:65) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) Caused by: java.lang.Exception: https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec-0.12.0.2.0.6.0-76/6.0.0/hive-exec-0.12.0.2.0.6.0-76-6.0.0.jar at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:119) at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:85) at org.talend.librariesmanager.nexus.utils.NexusDownloader.download(NexusDownloader.java:157) at org.talend.core.download.DownloadHelperWithProgress.download(DownloadHelperWithProgress.java:90) at org.talend.librariesmanager.utils.nexus.NexusDownloadHelperWithProgress.download(NexusDownloadHelperWithProgress.java:95) at org.talend.librariesmanager.utils.DownloadModuleRunnable.downLoad(DownloadModuleRunnable.java:110) ... 5 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.talend.librariesmanager.nexus.utils.NexusDownloader$1.execute(NexusDownloader.java:100) at org.talend.core.nexus.HttpClientTransport.doRequest(HttpClientTransport.java:112) ... 10 more
I'm still investigating. Currently I dought insuficcent timeout had set at HttpClientTransport#doRequest().
Any suggestions are welcome.
Regards
minek

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
In Studio, please navigate to Window > Preferences > Talend > Performance to increase timeout value to see if it is OK with you.
Best regards
Sabrina

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Thanks to reply.
I need some clarification. Did you mean "timeout value" is this pages "Connection timeout (seconds)" that is activated by to check "Activate the timeout for database connection." ? If so, the setting seems to be nothing to do for this case when I tried.
Regards
minek

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Does this issue happen all the time or only happens very often when download all Optional third-party libraries at one time?
The default socket read timeout is 60s, there is one way is to increase the readtimeout parameters in the studio ini file
-Dsun.net.client.defaultReadTimeout=600000
Is it OK with you?
Best regards
Sabrina

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
> Does this issue happen all the time or only happens very often when download all Optional third-party libraries at one time?
I'd tested "Install Additional Packages" several times I heard that.
- First time, Optional modules is 1135. then ended by timeout.
- 1134, timeout
- 1132, timeout
- 1130, timeout
- 1129, timeout
Tommorow, I will test with an fresh copy of 7.1.1.
-Dsun.net.client.defaultReadTimeout=600000
Does it mean 600,000 millsecs = 600 secs = 10mins ?
I just copy and paste the line in my studio.ini, but "timeout" errors didn't take a time I saw.
Regard
minek

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I had a little time today. so I focused on "install Additional Packages" function with a fresh copy. And it's not finished yet, but
Required modules - no error.
Optional modules - errors at
- (1140) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1139) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1138) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1137) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1133) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1131) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1129) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1128) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1126) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1126) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1125) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1122) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (1121) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/hive-exec...
- (493) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/pig-0.12....
- (417) https://talend-update.talend.com/nexus/content/repositories/libraries/org/talend/libraries/spark-ass...
Errors at similar modules yesterday. But I haven't check those are exact same.
May be I don't need understand this phenomenon, But I need the "retry download on failure" functionality.
Regards
minek

- « Previous Replies
-
- 1
- 2
- Next Replies »