Talend DI 6.4.1 Java routine accessing S3 failed with "method not found" error

Seven Stars

Talend DI 6.4.1 Java routine accessing S3 failed with "method not found" error

Hello everyone

 

I have a problem calling custom Java method which accesses AWS S3 (NoSuchMethodError)

 

Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.SDKGlobalConfiguration.isInRegionOptimizedModeEnabled()Z
at com.amazonaws.ClientConfigurationFactory.getConfig(ClientConfigurationFactory.java:35)
at com.amazonaws.client.builder.AwsClientBuilder.resolveClientConfiguration(AwsClientBuilder.java:163)
at com.amazonaws.client.builder.AwsClientBuilder.access$000(AwsClientBuilder.java:52)
at com.amazonaws.client.builder.AwsClientBuilder$SyncBuilderParams.<init>(AwsClientBuilder.java:411)
at com.amazonaws.client.builder.AwsClientBuilder.getSyncClientParams(AwsClientBuilder.java:354)
at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
at insightsanalytics.TransferPostgreDynSqlRdsToS3.copyPgSqlToS3(TransferPostgreDynSqlRdsToS3.java:272)
at insightsanalytics.TransferPostgreDynSqlRdsToS3.copyPgSqlToS3(TransferPostgreDynSqlRdsToS3.java:154)
at unpackelasticsearchjson.copydatafrompostgresqlrdstoredshift_0_1.CopyDataFromPostgreSQLRDStoRedshift.tPostgresqlInput_1Process(CopyDataFromPostgreSQLRDStoRedshift.java:855)

 

--

The routine is tested and works outside of Talend. 

When I call it inside tJava then I get above error:

I copied the AWS SDK jar into the Talend area and put the external routine dependency

I also explicitly imported the following in tJava:

 

import insightsanalytics.TransferPostgreDynSqlRdsToS3; // my code
import com.amazonaws.services.s3.*;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.BasicSessionCredentials;
import com.amazonaws.AmazonClientException;
import com.amazonaws.services.s3.model.*;

 

--

I have run out of ideas, please advise, given Talend DI generates pure Java, is there is a way to get it working ??

 

Many thanks!

 

 

Seven Stars

Re: Talend DI 6.4.1 Java routine accessing S3 failed with "method not found" error

More information.

 

I managed to get it to work by the following steps:

 

In Eclipse I picked the Export option Extract a Runnable JAR file (instead of plain Jar file) with a sub-option of Library handling: Extract required libraries into generated JAR file.

 

So I have a self-contained, rather huge Jar file of ~200 MB.

 

This worked! I tried both inside DI Studio and also ran generated bat file in Widows shell. Both worked.

 

So my question now is this - am I doing this correctly?

 

If there a better method to get external Java methods to work from Talend ?

My custom code also references other libraries such as AWS PostgreSQL, AWS Redshift, in addition to AWS S3.

The versions of libraries I use in my custom code and those AWS uses will likely be different to the libraries used by Talend DI.

I wonder if there is likely to be a conflict?

Example: Talend job may use PostgreSQL and Redshift itself and my custom Java may also use the same libraries.

 

Perhaps creating single large Runnable Jar with libraries packaged inside will somehow insulate/isolate my code and the library versions I use from those used by Talend DI?

PS: The build job task generates a massive zip file of ~500 MB (which includes my custom JAR of 200+ MB). I am not very concerned about the size but more about recommended best practices.

 

Many thanks everyone

 

 

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now