Talend ESB: using and managing different versions of Amazon SQS Java SDK with in Talend.

Six Stars

Talend ESB: using and managing different versions of Amazon SQS Java SDK with in Talend.



I am using Talend Data Fabric 6.4.1 version. 

I have the requirement to use Amazon SQS FIFO queues. Camel SQS component (cAWSSQS) doesn't support FIFO queues yet. i.e I am not able to pass the messageGroupId when I enqueue messages.

So, I've decided to write a bean that can handle enqueing  messages to a FIFO queue using aws-java-sdk-sqs-1.11.204.jar. 

Talend is using aws-java-sdk-sqs-1.10.52 version for cAWSSQS component. I had to use a higher version in the bean's code, as 1.10.52 doesn't have support for FIFO queues. In the bean's "Edit bean libraries" option, I had to reference 1.11.204 version of sqs as well as all its transient dependencies explicitly for it to work. I also added it to the modules list ( see attached screenshot for modules. ) as part of my work, but unable to uninstall it now. 


I want to use the custom code only for enqueing. But in a separate route, I want to use Talend's cAWSSQS component for dequeue messages.

Problem here is, when I run the route that is using cAWSSQS to dequeue messages,  higher version of dependency libraries are loaded and run is failing with the below error. 

I tried clearing the .java/lib folder, still both version of the library are loaded when I run this route that simply uses cAWSSQS component. How can I have different routes using different version of the same library ( and its transient dependencies)? is that doable? please suggest. 


Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifierSmiley WinkV
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.<init>(SdkTLSSocketFactory.java:56)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:50)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:38)
at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:315)
at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:299)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:164)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:153)





Six Stars

Re: Talend ESB: using and managing different versions of Amazon SQS Java SDK with in Talend.

Now, I understand why the Deqeueu route uses the higher version. routines.jar is loaded by default for this route, and routines.jar includes all the beans. 


To avoid all this mess, I am trying to see if I can write a routelet that can Enqueue messages using my custom code. But I am not able to drag and drop the routelet to any route. It says "Cannot apply drag and drop operation on this component currently". This could be because I am loading some libraries using cConfig component with in the routelet. Please suggest if this is doable or not. Where else can I keep my custom code, so that is is disconnected from other routes but only available in the references routes.




Talend named a Leader.

Get your copy


Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Talend API Designer – Technical Overview

Take a look at this technical overview video of Talend API Designer

Watch Now

Getting Started with APIs

Find out how to get started with APIs