Talend ESB: using and managing different versions of Amazon SQS Java SDK with in Talend.

Highlighted
Six Stars

Talend ESB: using and managing different versions of Amazon SQS Java SDK with in Talend.

Hi,

 

I am using Talend Data Fabric 6.4.1 version. 

I have the requirement to use Amazon SQS FIFO queues. Camel SQS component (cAWSSQS) doesn't support FIFO queues yet. i.e I am not able to pass the messageGroupId when I enqueue messages.

So, I've decided to write a bean that can handle enqueing  messages to a FIFO queue using aws-java-sdk-sqs-1.11.204.jar. 

Talend is using aws-java-sdk-sqs-1.10.52 version for cAWSSQS component. I had to use a higher version in the bean's code, as 1.10.52 doesn't have support for FIFO queues. In the bean's "Edit bean libraries" option, I had to reference 1.11.204 version of sqs as well as all its transient dependencies explicitly for it to work. I also added it to the modules list ( see attached screenshot for modules. ) as part of my work, but unable to uninstall it now. 

 

I want to use the custom code only for enqueing. But in a separate route, I want to use Talend's cAWSSQS component for dequeue messages.

Problem here is, when I run the route that is using cAWSSQS to dequeue messages,  higher version of dependency libraries are loaded and run is failing with the below error. 

I tried clearing the .java/lib folder, still both version of the library are loaded when I run this route that simply uses cAWSSQS component. How can I have different routes using different version of the same library ( and its transient dependencies)? is that doable? please suggest. 

 

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.conn.ssl.SSLConnectionSocketFactory.<init>(Ljavax/net/ssl/SSLContext;Ljavax/net/ssl/HostnameVerifierSmiley WinkV
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.<init>(SdkTLSSocketFactory.java:56)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.getPreferredSocketFactory(ApacheConnectionManagerFactory.java:92)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:65)
at com.amazonaws.http.apache.client.impl.ApacheConnectionManagerFactory.create(ApacheConnectionManagerFactory.java:58)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:50)
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:38)
at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:315)
at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:299)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:164)
at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:153)

 

thanks,

Sowjanya.

 

Six Stars

Re: Talend ESB: using and managing different versions of Amazon SQS Java SDK with in Talend.

Now, I understand why the Deqeueu route uses the higher version. routines.jar is loaded by default for this route, and routines.jar includes all the beans. 

 

To avoid all this mess, I am trying to see if I can write a routelet that can Enqueue messages using my custom code. But I am not able to drag and drop the routelet to any route. It says "Cannot apply drag and drop operation on this component currently". This could be because I am loading some libraries using cConfig component with in the routelet. Please suggest if this is doable or not. Where else can I keep my custom code, so that is is disconnected from other routes but only available in the references routes.

 

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Talend API Designer – Technical Overview

Take a look at this technical overview video of Talend API Designer

Watch Now

Getting Started with APIs

Find out how to get started with APIs

Read