[ESB] How to handle large files with ActiveMQ ?

Seven Stars

[ESB] How to handle large files with ActiveMQ ?

Hi

In a esb route, a messagingEndpoint reads a large file and sends it to an AMQ queue. But it fails if the file is larger than 50Mb, wich is not very large itself.

The error is : "Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Socket closed", I think it's a timeout that occurs after approximatly 30 seconds.

 

What parameters or other way can we use to handle large files in Talend ESB and ActiveMQ (seeded with Talend ESB) ?

 

Thanks

Loko


Accepted Solutions
Seven Stars

Re: [ESB] How to handle large files with ActiveMQ ?

Just installed Artemis and it works very well. No Talend modification needed. 1 GB file produced and consumed on my dev box in less than 1sec.


All Replies
Sixteen Stars

Re: [ESB] How to handle large files with ActiveMQ ?

Have you looked at this documentation? 

https://activemq.apache.org/artemis/docs/1.0.0/large-messages.html

 

You may want to use a cMessagingEndpoint and use the Apache Camel JMS component from scratch. See this:

http://camel.apache.org/jms.html

Seven Stars

Re: [ESB] How to handle large files with ActiveMQ ?

Hi

Yes I've read this doc but it is for Artemis, not standard ActiveMQ seeded with Talend ESB.

 

I'll try the basic jms endpoint but not convinced it will change anything as the Talend component does more or less the same.

Sixteen Stars

Re: [ESB] How to handle large files with ActiveMQ ?

The cMessagingEndpoint generally allows more flexibility as you have complete control over the Apachel Camel component. Talend's "wrapper" components, make life a lot easier, but sometimes it gives you more control to experiment when using the plain Apache Camel stuff.

Seven Stars

Re: [ESB] How to handle large files with ActiveMQ ?

we're using cMessaingEndpoint for convenience use but you can achieve the same with Talend component if you put the camel options in the "advanced settings'".

 

Anyway, I'm trying this

Sixteen Stars

Re: [ESB] How to handle large files with ActiveMQ ?

That is true in most scenarios, but I've just found it easier to use the Apache Camel components' full URIs. As I said, it just gives you complete control and allows you to focus on one set of documentation instead of cross-referencing when the functionality is implemented in Talend without using the Advanced Settings. 

Employee Ram
Employee

Re: [ESB] How to handle large files with ActiveMQ ?

For cMessageEndpoint try adding requestTimeout to URI Ex:?option1=value&option2=value

Seven Stars

Re: [ESB] How to handle large files with ActiveMQ ?

I can't find the full JMS URI syntax , it failed retrived the cMQconnectionFactory I add as parameter, while I can see it in the java code. I tried the java object Talend creates and also the name of the object it adds in the Camel context but it fails with "No bean could be found in the registry for: myfactory of type: javax.jms.ConnectionFactory"

 

With cJMS component, I tried this parameter , adding a cExchangePattern INOUT before as the doc says the parameter only works this way, but it does not seem to have any effect, the timeout does not occur later. In the java code I can see :

".to("myfactory:queue:myMQ?requestTimeout=2000000"

 

I also tried Transacted mode in factory and in producer with "&transactionTimeout=120" , no effect either.

Sixteen Stars

Re: [ESB] How to handle large files with ActiveMQ ?

You're better off using Connection Pooling for JMS in the runtime (Karaf) and you can emulate this using the Spring tab in the Studio.

 

https://help.talend.com/reader/8avcOyufaVYo7Uoah0kOlA/hwTS2VEbUI1xln~zxK2Kyg 

Seven Stars

Re: [ESB] How to handle large files with ActiveMQ ?

Just installed Artemis and it works very well. No Talend modification needed. 1 GB file produced and consumed on my dev box in less than 1sec.