Routes and Error Handling

Seven Stars

Routes and Error Handling

Hello

 

A couple of quick questions about handling errors in messaging routes. I'm hopeful that the answers are quick too. Smiley Happy

 

1) Manually fixing XML messages in flight.

Wondering if, and how people approach this. Do people store invalid msgs in a db and fix the xml there or go onto the Dead Letter Queue and fix the xml in ActiveMQ and move the message back?

 

2) Redelivery of fixed invalid messages, when order of the messages is vital. 

(for example message 1 is "order item" and message 2 is "cancel item")

Do people:

 

i) block message 2, maybe based on some (group) id and replay both messages, in order, when message 1 is fixed? If so are people using ActiveMQ or DBs to store and manage this ordered redelivery?

ii) do something else? If so - what would that be?

 

Apologies if my questions are too high level in the first instance...

 

Cheers

 

 

 

 

 

Community Manager

Re: Routes and Error Handling

How is the XML broken? If it is not valid XML how will know which message it is and what it is related to without String parsing? The best way to deal with this is to ensure that the XML is not broken before it is picked up by the route. If you are dealing with data errors rendering the business logic of your XML broken, then you will need to add another layer. I cannot see a way of maintaining your strict processing order if you are not aware of every message that will be sent. For example, if you have the options of...

 

Order Item
Cancel Item
Modify Item

 

....you can implement some logic to deal with some scenarios. For example, you cannot cancel an order that does not exist and you cannot modify an order that doesn't exist. However, if you have your order received OK, how will you know that a Modify should have arrived before a Cancel, for example? If the Modify fails, the Cancel could cause issues. I suspect that you might need a "holding" status for Orders for which there are errors in processing (if you can get this data). So when a message comes in that is broken for whatever reason, a hold is put on processing all subsequent messages for that order. The held messages would have to be stored somewhere while the broken message is fixed. Then once the broken message is processed, all subsequent messages could be released. This will be quite tricky to implement

Seven Stars

Re: Routes and Error Handling

Hello

Many thanks for the response.

Yes it is tricky and maybe we are over engineering a problem in advance of experiencing it.

 

It sounds like fixing messages in flight is not something that is done per se? Some esbs like biztalk seem to offer this functionality out of the box.https://docs.microsoft.com/en-us/biztalk/esb-toolkit/collecting-exceptions-and-routing-messages-for-......but ActiveMQ and JMS say messages are immutable once they have been sent.

 

We have quite disparate systems that fire and forget, so data structures could change and we could get messages that the ESB deems as invalid and they need to go to a invalid message channel for manual inspection. If there is a required order to messages and some still are valid then we were wondering how to deal with maintaining a strict order.

 

Like you say building business and application logic to guard against things getting out of order sounds the way to go. 

 

I'm just curious if there is any OOTB functionality we can make use of with talend and ActiveMQ.

 

Googling brings up concepts of ActiveMQ transactions, rollback, Message Groups to group messages together, Units of Work in Camel or other MQs like Websphere, strictOrderDispatchPolicies and blocking delivery, etc.

 

But I'm not sure if/how these concepts could be pulled together to help us?

 

Cheers

 

Community Manager

Re: Routes and Error Handling

Sorry, I may have misunderstood your requirement. When it comes to fixing messages in flight, I don't see how Biztalk can do that. If it is expecting .....

 

<data>
<name>Richard Hall</name>
<age>40</age>
</data>

....but receives....

<data>
<job>Community Manager</job>
<location>London</location>
</data>

..... it cannot possibly fix it automatically.

 

If you know the sort of issues you will get with your messages, then you can certainly build "fix in flight" into your routes with Talend. However I sense that something like this might be what you want https://help.talend.com/reader/1JiA7GVBIG_vkniH3yI8Mg/NqAMavU7YFHOqi6c9qkdbA

 

If you can define what you are looking for in a good message, you can use the example in the above link to help you send messages to an error queue or similar. 

 

With regard to dealing with other messages from the same order, you will still have the issues I mentioned in my last post.

Eight Stars

Re: Routes and Error Handling

That's a pity AMQ messages are immutable Smiley Sad

Thus, we decided to send wrong messages by email attachements to accurate user , then I can fix it and post it again in a directory --> cMessagingEndpoint --> processed again by the route. 

 

Kafka could be a workaround, it offers easier replays for messages

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

An API-First Approach to Modernizing Applications

Learn how to use an API-First Approach to Modernize your Applications

Watch Now

Talend API Designer – Technical Overview

Take a look at this technical overview video of Talend API Designer

Watch Now

Getting Started with APIs

Find out how to get started with APIs

Read