Talend DI 6.4.1 Java routine accessing S3 failed with "method not found" error

Anonymous — Sat, 16 Nov 2024 09:10:41 GMT

Hello everyone

I have a problem calling custom Java method which accesses AWS S3 (NoSuchMethodError)

Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.SDKGlobalConfiguration.isInRegionOptimizedModeEnabled()Z
at com.amazonaws.ClientConfigurationFactory.getConfig(ClientConfigurationFactory.java:35)
at com.amazonaws.client.builder.AwsClientBuilder.resolveClientConfiguration(AwsClientBuilder.java:163)
at com.amazonaws.client.builder.AwsClientBuilder.access$000(AwsClientBuilder.java:52)
at com.amazonaws.client.builder.AwsClientBuilder$SyncBuilderParams.<init>(AwsClientBuilder.java:411)
at com.amazonaws.client.builder.AwsClientBuilder.getSyncClientParams(AwsClientBuilder.java:354)
at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at insightsanalytics.TransferPostgreDynSqlRdsToS3.copyPgSqlToS3(TransferPostgreDynSqlRdsToS3.java:272)
at insightsanalytics.TransferPostgreDynSqlRdsToS3.copyPgSqlToS3(TransferPostgreDynSqlRdsToS3.java:154)
at unpackelasticsearchjson.copydatafrompostgresqlrdstoredshift_0_1.CopyDataFromPostgreSQLRDStoRedshift.tPostgresqlInput_1Process(CopyDataFromPostgreSQLRDStoRedshift.java:855)

The routine is tested and works outside of Talend.

When I call it inside tJava then I get above error:

I copied the AWS SDK jar into the Talend area and put the external routine dependency

I also explicitly imported the following in tJava:

import insightsanalytics.TransferPostgreDynSqlRdsToS3; // my code
import com.amazonaws.services.s3.*;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.BasicSessionCredentials;
import com.amazonaws.AmazonClientException;
import com.amazonaws.services.s3.model.*;

I have run out of ideas, please advise, given Talend DI generates pure Java, is there is a way to get it working ??

Many thanks!

Re: Talend DI 6.4.1 Java routine accessing S3 failed with "method not found" error

Anonymous — Fri, 13 Oct 2017 02:15:25 GMT

More information.

I managed to get it to work by the following steps:

In Eclipse I picked the Export option Extract a Runnable JAR file (instead of plain Jar file) with a sub-option of Library handling: Extract required libraries into generated JAR file.

So I have a self-contained, rather huge Jar file of ~200 MB.

This worked! I tried both inside DI Studio and also ran generated bat file in Widows shell. Both worked.

So my question now is this - am I doing this correctly?

If there a better method to get external Java methods to work from Talend ?

My custom code also references other libraries such as AWS PostgreSQL, AWS Redshift, in addition to AWS S3.

The versions of libraries I use in my custom code and those AWS uses will likely be different to the libraries used by Talend DI.

I wonder if there is likely to be a conflict?

Example: Talend job may use PostgreSQL and Redshift itself and my custom Java may also use the same libraries.

Perhaps creating single large Runnable Jar with libraries packaged inside will somehow insulate/isolate my code and the library versions I use from those used by Talend DI?

PS: The build job task generates a massive zip file of ~500 MB (which includes my custom JAR of 200+ MB). I am not very concerned about the size but more about recommended best practices.

Many thanks everyone

topic Talend DI 6.4.1 Java routine accessing S3 failed with "method not found" error in Talend Studio

Talend DI 6.4.1 Java routine accessing S3 failed with "method not found" error

Re: Talend DI 6.4.1 Java routine accessing S3 failed with "method not found" error