Talend Open Studio v. 7.1.1 cannot download additional packages

[msminek]

Hi,

The  "Install Additional Packages" and the download functionality in the module view didn't work in my Talend Open Studio for Data Integration v7.1.1.
I  found some important files under https://update.talend.com/Studio/tos/7.1.1/ are disappeard. It seems to be a problem.
How can I fix it ? 

Regards
minek

[msminek]

Hi,

I've done the test. Finally I've got a all optional modules when I 've called "install Additional Packages" 33 times for the test. It's terrible result.

$ grep "ERROR org.talend.commons.exception.CommonExceptionHandler  - Download " < ".log" 
!MESSAGE 2018-11-15 17:56:59,228 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-1.2.0-mapr-1608.jar failed!
!MESSAGE 2018-11-15 18:00:09,460 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-1.2.0-mapr-1710.jar failed!
!MESSAGE 2018-11-15 18:01:18,061 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-1.2.0-mapr-1803-spark.jar failed!
!MESSAGE 2018-11-15 18:02:56,299 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-1.2.0-mapr.jar failed!
!MESSAGE 2018-11-15 18:04:27,564 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-1.2.1000.2.4.0.0-169.jar failed!
!MESSAGE 2018-11-15 18:05:26,754 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-1.2.1000.2.5.0.0-1245.jar failed!
!MESSAGE 2018-11-15 18:06:27,731 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-1.2.1000.2.6.0.3-8.jar failed!
!MESSAGE 2018-11-15 18:07:18,328 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-2.1.0-amzn-0.jar failed!
!MESSAGE 2018-11-15 18:08:27,256 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-2.1.1-amzn-0-talend-nolang3.jar failed!
!MESSAGE 2018-11-15 18:09:11,085 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-2.1.1-amzn-0-talend-nolang3.jar failed!
!MESSAGE 2018-11-15 18:10:28,698 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-2.1.1-amzn-0.jar failed!
!MESSAGE 2018-11-15 18:12:18,444 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-2.3.0-amzn-0.jar failed!
!MESSAGE 2018-11-15 18:13:09,387 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download hive-exec-2.3.3-amzn-0.jar failed!
!MESSAGE 2018-11-15 18:32:54,300 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download pig-0.12.0-cdh5.12.1.jar failed!
!MESSAGE 2018-11-15 18:46:17,451 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-assembly-1.5.2-mapr-1602-hadoop2.7.0-mapr-1602.jar failed!
!MESSAGE 2018-11-15 18:48:36,219 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar failed!
!MESSAGE 2018-11-15 19:02:24,581 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-assembly-1.6.2.2.5.0.0-1245-hadoop2.7.3.2.5.0.0-1245.jar failed!
!MESSAGE 2018-11-15 19:09:56,467 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-hive-exec_2.11-2.1.0.cloudera1.jar failed!
!MESSAGE 2018-11-15 19:18:54,464 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-sql_2.11-2.1.0.jar failed!
!MESSAGE 2018-11-15 19:23:33,274 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-streaming-kafka-assembly_2.10-1.4.1.2.3.2.0-2950.jar failed!
!MESSAGE 2018-11-15 19:32:25,507 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-streaming-kafka-assembly_2.10-1.6.3.2.6.0.3-8.jar failed!
!MESSAGE 2018-11-15 19:33:50,785 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-streaming-kinesis-asl-assembly_2.10-1.5.0.jar failed!
!MESSAGE 2018-11-15 19:35:16,616 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-streaming-kinesis-asl-assembly_2.10-1.6.0-cdh5.10.1.jar failed!
!MESSAGE 2018-11-15 19:36:43,834 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-streaming-kinesis-asl-assembly_2.10-1.6.1.jar failed!
!MESSAGE 2018-11-15 19:38:18,659 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download spark-streaming-kinesis-asl-assembly_2.11-2.1.0.cloudera1.jar failed!
!MESSAGE 2018-11-15 19:41:38,711 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download talend-spark-assembly-1.3.1-hadoop2.5.1-mapr-1501.jar failed!
!MESSAGE 2018-11-15 19:43:59,660 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download talend-spark-assembly-1.4.1.2.3.2.0.jar failed!
!MESSAGE 2018-11-15 19:45:25,246 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download talend-spark-assembly-1.5.0-cdh5.5.2-hadoop2.6.0-with-hive-07122016.jar failed!
!MESSAGE 2018-11-15 19:47:44,385 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download talend-spark-assembly-1.6.0-cdh5.8.1-hadoop2.6.0-cdh5.8.1-with-hive.jar failed!
!MESSAGE 2018-11-15 19:50:12,766 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download talend-spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar failed!
!MESSAGE 2018-11-15 19:55:42,676 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download talend-spark-assembly-1.6.1-hadoop2.7.2-amzn-0.jar failed!
!MESSAGE 2018-11-15 19:57:42,849 ERROR org.talend.commons.exception.CommonExceptionHandler  - Download talend-spark-assembly-1.6.1-mapr-1608-hadoop2.7.0-mapr-1607.jar failed!

I still want a someone to push these buttons  for me :-).

Regards

minek

[Anonymous]

Hello,

We cannot repro this issue on talend open studio V 7.1.1, window 10.

Is your internet connection Ok with you?

Best regards

Sabrina

[msminek · Accepted solution]

Hi,

Thank you for your testing. Yes, maybe you are right. There might be a bit tricky network traffic pattern caused by the proxy server, the vaccine solution or so on.

I'm writing the auto retry hack for the tcommon-studio-se project. It's almost done. Code is below. I will go to jila later.

And I also find the timeout setting in the downloding from nexus. NexusServerUtils.getTimeout() tells me that if nothing set in the preference page, the default value is 20000 millisecs. If we have the preference page for the value, this problem will be solved - or still needs retrying.

https://github.com/Talend/tcommon-studio-se/blob/master/main/plugins/org.talend.core.runtime/src/mai...

diff --git a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
index 7d023dfd40..58a3298dda 100644
--- a/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
+++ b/main/plugins/org.talend.core.runtime/src/main/java/org/talend/core/nexus/HttpClientTransport.java
@@ -86,6 +86,41 @@ public abstract class HttpClientTransport {
     }
 
     public void doRequest(IProgressMonitor monitor, final URI requestURI) throws Exception {
+        int retries = 5;
+        long waitMillis = 20000;
+
+        int timeout = NexusServerUtils.getTimeout();
+        boolean fTimeout = false;
+
+        for(int t = 1; t <= retries; t++){
+            try{
+                fTimeout = false;
+                doRequestOne(monitor, requestURI);
+            } catch (java.net.SocketTimeoutException e){
+                // Read timed out
+                fTimeout = true;
+                if(t == retries){
+                    throw new Exception(e);
+                }
+            } catch (Exception e) {
+                // DEBUG //
+                System.err.printf("[%d] Exception occured for %s\n", t, requestURI);
+                e.printStackTrace(System.err);
+                // DEBUG //
+                throw e;
+            }
+            if(!fTimeout){
+                break;
+            }
+            // DEBUG //
+            System.err.printf("[%d] Read timeout (in %d millisecs) occured for %s\n", t, timeout, requestURI);
+            // DEBUG //
+
+            Thread.sleep(waitMillis * t);
+        }
+    }
+
+    public void doRequestOne(IProgressMonitor monitor, final URI requestURI) throws Exception {
         if (monitor == null) {
             monitor = new NullProgressMonitor();
         }
@@ -115,6 +150,9 @@ public abstract class HttpClientTransport {
         } catch (org.apache.http.conn.HttpHostConnectException e) {
             // connection failure
             throw e;
+        } catch (java.net.SocketTimeoutException e){
+            // Read timed out
+            throw e;
         } catch (Exception e) {
             throw new Exception(requestURI.toString(), e);
         } finally {
@@ -147,13 +185,15 @@ public abstract class HttpClientTransport {
                         address.getAddress(), proxyPort, "Http Proxy", "Http proxy authentication", null);
                 if (proxyAuthentication != null) {
                     String proxyUser = proxyAuthentication.getUserName();
-                    String proxyPassword = "";
-                    char[] passwordChars = proxyAuthentication.getPassword();
-                    if (passwordChars != null) {
-                        proxyPassword = new String(passwordChars);
+                    if(StringUtils.isNotBlank(proxyUser)){
+                        String proxyPassword = "";
+                        char[] passwordChars = proxyAuthentication.getPassword();
+                        if (passwordChars != null) {
+                            proxyPassword = new String(passwordChars);
+                        }
+                        httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort),
+                                new UsernamePasswordCredentials(proxyUser, proxyPassword));
                     }
-                    httpClient.getCredentialsProvider().setCredentials(new AuthScope(proxyServer, proxyPort),
-                            new UsernamePasswordCredentials(proxyUser, proxyPassword));
                 }
                 HttpHost proxyHost = new HttpHost(proxyServer, proxyPort);
                 httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxyHost);
diff --git a/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java b/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java
index 5f62a9bc6e..f190ce3c2c 100644
--- a/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java
+++ b/main/plugins/org.talend.librariesmanager/src/main/java/org/talend/librariesmanager/nexus/utils/NexusDownloader.java
@@ -91,6 +91,7 @@ public class NexusDownloader implements IDownloadHelper {
 
             NullProgressMonitor monitor = new NullProgressMonitor();
             ArtifactRepositoryBean nServer = getNexusServer();
+            try{
             new HttpClientTransport(nServer.getRepositoryURL(), nServer.getUserName(), nServer.getPassword()) {
 
                 @Override
@@ -145,9 +146,6 @@ public class NexusDownloader implements IDownloadHelper {
                             if (bos != null) {
                                 bos.close();
                             }
-                            if (tempFolder != null) {
-                                FilesUtils.deleteFile(tempFolder, true);
-                            }
                         }
 
                     }
@@ -155,6 +153,11 @@ public class NexusDownloader implements IDownloadHelper {
                 }
 
             }.doRequest(monitor, parseMvnUrl);
+            }finally{
+                if (tempFolder != null) {
+                    FilesUtils.deleteFile(tempFolder, true);
+                }
+            }
         }
 
     }

Thank you for your cooperating.

Regards,

minek

[Anonymous]

Hi, I am having the same problem. I am not able to install external module. I can see the Download and Install button and it get away when I click on. After that the nothing happens, neither module downloading nor any error message. I have tried with Talend 7.0.1 and 7.1.1 . I am using Windows 10 with Java 1.8

Please see the error logs-

====================================================
eclipse.buildId=unknown
java.version=1.8.0_181
java.vendor=Oracle Corporation
BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=en_US
Command-line arguments: -os win32 -ws win32 -arch x86_64

!ENTRY org.eclipse.jdt.launching.macosx 4 0 2019-01-17 21:42:33.351
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.jdt.launching.macosx [334]
Unresolved requirement: Require-Capability: eclipse.platform; filter:="(osgi.os=macosx)"

at org.eclipse.osgi.container.Module.start(Module.java:434)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1582)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1561)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1533)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1476)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)

!ENTRY org.eclipse.jdt.launching.ui.macosx 4 0 2019-01-17 21:42:33.355
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.jdt.launching.ui.macosx [336]
Unresolved requirement: Require-Capability: eclipse.platform; filter:="(osgi.os=macosx)"

at org.eclipse.osgi.container.Module.start(Module.java:434)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1582)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1561)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1533)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1476)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340)

!ENTRY org.eclipse.core.resources 2 10035 2019-01-17 21:42:34.572
!MESSAGE The workspace exited with unsaved changes in the previous session; refreshing workspace to recover changes.

!ENTRY org.talend.platform.logging 1 0 2019-01-17 21:42:36.895
!MESSAGE 2019-01-17 21:42:36,893 INFO org.talend.repository.ui.login.LoginProjectPage - validate updatesite...

!ENTRY org.talend.platform.logging 2 0 2019-01-17 21:42:40.997
!MESSAGE 2019-01-17 21:42:40,997 WARN org.talend.sdk.component.studio.logging.JULToOsgiHandler - [org.apache.webbeans.config.BeansDeployer.configureInterceptors] Interceptor class : org.apache.deltaspike.core.impl.throttling.ThrottledInterceptor is already defined

!ENTRY org.talend.platform.logging 2 0 2019-01-17 21:42:41.006
!MESSAGE 2019-01-17 21:42:41,005 WARN org.talend.sdk.component.studio.logging.JULToOsgiHandler - [org.apache.webbeans.config.BeansDeployer.configureInterceptors] Interceptor class : org.apache.deltaspike.core.impl.lock.LockedInterceptor is already defined

!ENTRY org.talend.platform.logging 2 0 2019-01-17 21:42:41.013
!MESSAGE 2019-01-17 21:42:41,013 WARN org.talend.sdk.component.studio.logging.JULToOsgiHandler - [org.apache.webbeans.config.BeansDeployer.configureInterceptors] Interceptor class : org.apache.deltaspike.core.impl.future.FutureableInterceptor is already defined

!ENTRY org.talend.platform.logging 2 0 2019-01-17 21:42:41.710
!MESSAGE 2019-01-17 21:42:41,709 WARN org.talend.sdk.component.studio.logging.JULToOsgiHandler - [org.apache.cxf.jaxrs.impl.ConfigurableImpl.doRegister] Null, empty or invalid contracts specified for brave.jaxrs2.TracingClientFilter@57877880; ignoring.

!ENTRY org.talend.platform.logging 1 0 2019-01-17 21:42:44.174
!MESSAGE 2019-01-17 21:42:44,174 INFO org.talend.core.repository.model.ProxyRepositoryFactory - user@talend.com logged on MatSpec

!ENTRY org.talend.platform.logging 1 0 2019-01-17 21:42:45.203
!MESSAGE 2019-01-17 21:42:45,203 INFO org.talend.core.repository.logintask.ReadOnlyProjectConfirm - ReadOnly confirm

!ENTRY org.talend.platform.logging 1 0 2019-01-17 21:42:46.087
!MESSAGE 2019-01-17 21:42:46,087 INFO org.eclipse.m2e.internal.launch.MavenLaunchDelegate - D:\Siemens\TOS_DI-20180411_1414-V7.0.1\workspace\MATSPEC\poms\code\routines

!ENTRY org.talend.platform.logging 1 0 2019-01-17 21:42:46.095
!MESSAGE 2019-01-17 21:42:46,095 INFO org.eclipse.m2e.internal.launch.MavenLaunchDelegate - mvn -B -o -s D:\Siemens\TOS_DI-20180411_1414-V7.0.1\configuration\maven_user_settings.xml install -llr -Dmaven.main.skip=true -P !ci-builder

!ENTRY org.talend.platform.logging 1 0 2019-01-17 21:42:48.952
!MESSAGE 2019-01-17 21:42:48,952 INFO org.eclipse.m2e.internal.launch.MavenLaunchDelegate - D:\Siemens\TOS_DI-20180411_1414-V7.0.1\workspace\MATSPEC\poms\code\routines

!ENTRY org.talend.platform.logging 1 0 2019-01-17 21:42:48.979
!MESSAGE 2019-01-17 21:42:48,978 INFO org.eclipse.m2e.internal.launch.MavenLaunchDelegate - mvn -B -o -s D:\Siemens\TOS_DI-20180411_1414-V7.0.1\configuration\maven_user_settings.xml install -llr -Dmaven.main.skip=true -P !ci-builder

!ENTRY org.eclipse.ui 2 2 2019-01-17 21:43:03.807
!MESSAGE Invalid preference category path: org.talend.designer.runtime.visualization.internal.ui.JavaMonitorPreferencePage (bundle: org.talend.designer.runtime.visualization.tools, page: org.talend.designer.runtime.visualization.internal.tools.ToolsPreferencePage)

[Anonymous]

Hi,

Did you find any solution for this? I too have similar problem.

Thanks,

Santhosh

[Anonymous]

Not yet, I am installing modules manually mentioned here

https://community.talend.com/t5/Installing-and-Upgrading/I-can-t-download-third-party-jar-Files/m-p/....

Talend team is working on this issue and will update us soon. 

Thanks and Regards

[DREYNAUD]

Same error here on a Windows 10 / Java 8 64 bits fresh install.

It is tmpossible to test Talend Open Studio, it's frustracting, i hope it will be solved soon.

Best regards.

[msminek]

Hi all,

 FYI, I had confirmed to fix the problem in Talend Open Studio for Data Integration 7.2.1M1 at Feb. 4.

 But now, TOS for DI 7.2.1M2 has another module installation problem "Error resolving artifact graphframes ..." might be caused by the repository issue, I wish . So, I cannot check the problem fixed or not.

 And I also found Talend Open Studio for Data Quality has another or a same problem behind the proxy without authentication.

It's more frustrated than previous DI releases.

Best regards.