Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Save $650 on Qlik Connect, Dec 1 - 7, our lowest price of the year. Register with code CYBERWEEK: Register
cancel
Showing results for 
Search instead for 
Did you mean: 
Loko
Creator II
Creator II

[ESB] How to handle large files with ActiveMQ ?

Hi

In a esb route, a messagingEndpoint reads a large file and sends it to an AMQ queue. But it fails if the file is larger than 50Mb, wich is not very large itself.

The error is : "Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Socket closed", I think it's a timeout that occurs after approximatly 30 seconds.

 

What parameters or other way can we use to handle large files in Talend ESB and ActiveMQ (seeded with Talend ESB) ?

 

Thanks

Loko

Labels (3)
1 Solution

Accepted Solutions
Loko
Creator II
Creator II
Author

Just installed Artemis and it works very well. No Talend modification needed. 1 GB file produced and consumed on my dev box in less than 1sec.

View solution in original post

9 Replies
Anonymous
Not applicable

Have you looked at this documentation? 

https://activemq.apache.org/artemis/docs/1.0.0/large-messages.html

 

You may want to use a cMessagingEndpoint and use the Apache Camel JMS component from scratch. See this:

http://camel.apache.org/jms.html

Loko
Creator II
Creator II
Author

Hi

Yes I've read this doc but it is for Artemis, not standard ActiveMQ seeded with Talend ESB.

 

I'll try the basic jms endpoint but not convinced it will change anything as the Talend component does more or less the same.

Anonymous
Not applicable

The cMessagingEndpoint generally allows more flexibility as you have complete control over the Apachel Camel component. Talend's "wrapper" components, make life a lot easier, but sometimes it gives you more control to experiment when using the plain Apache Camel stuff.

Loko
Creator II
Creator II
Author

we're using cMessaingEndpoint for convenience use but you can achieve the same with Talend component if you put the camel options in the "advanced settings'".

 

Anyway, I'm trying this

Anonymous
Not applicable

That is true in most scenarios, but I've just found it easier to use the Apache Camel components' full URIs. As I said, it just gives you complete control and allows you to focus on one set of documentation instead of cross-referencing when the functionality is implemented in Talend without using the Advanced Settings. 

Anonymous
Not applicable

For cMessageEndpoint try adding requestTimeout to URI Ex:?option1=value&option2=value

Loko
Creator II
Creator II
Author

I can't find the full JMS URI syntax , it failed retrived the cMQconnectionFactory I add as parameter, while I can see it in the java code. I tried the java object Talend creates and also the name of the object it adds in the Camel context but it fails with "No bean could be found in the registry for: myfactory of type: javax.jms.ConnectionFactory"

 

With cJMS component, I tried this parameter , adding a cExchangePattern INOUT before as the doc says the parameter only works this way, but it does not seem to have any effect, the timeout does not occur later. In the java code I can see :

".to("myfactory:queue:myMQ?requestTimeout=2000000"

 

I also tried Transacted mode in factory and in producer with "&transactionTimeout=120" , no effect either.

Anonymous
Not applicable

You're better off using Connection Pooling for JMS in the runtime (Karaf) and you can emulate this using the Spring tab in the Studio.

 

https://help.talend.com/reader/8avcOyufaVYo7Uoah0kOlA/hwTS2VEbUI1xln~zxK2Kyg 

Loko
Creator II
Creator II
Author

Just installed Artemis and it works very well. No Talend modification needed. 1 GB file produced and consumed on my dev box in less than 1sec.