A couple of quick questions about handling errors in messaging routes. I'm hopeful that the answers are quick too.
1) Manually fixing XML messages in flight.
Wondering if, and how people approach this. Do people store invalid msgs in a db and fix the xml there or go onto the Dead Letter Queue and fix the xml in ActiveMQ and move the message back?
2) Redelivery of fixed invalid messages, when order of the messages is vital.
(for example message 1 is "order item" and message 2 is "cancel item")
i) block message 2, maybe based on some (group) id and replay both messages, in order, when message 1 is fixed? If so are people using ActiveMQ or DBs to store and manage this ordered redelivery?
ii) do something else? If so - what would that be?
Apologies if my questions are too high level in the first instance...
How is the XML broken? If it is not valid XML how will know which message it is and what it is related to without String parsing? The best way to deal with this is to ensure that the XML is not broken before it is picked up by the route. If you are dealing with data errors rendering the business logic of your XML broken, then you will need to add another layer. I cannot see a way of maintaining your strict processing order if you are not aware of every message that will be sent. For example, if you have the options of...
....you can implement some logic to deal with some scenarios. For example, you cannot cancel an order that does not exist and you cannot modify an order that doesn't exist. However, if you have your order received OK, how will you know that a Modify should have arrived before a Cancel, for example? If the Modify fails, the Cancel could cause issues. I suspect that you might need a "holding" status for Orders for which there are errors in processing (if you can get this data). So when a message comes in that is broken for whatever reason, a hold is put on processing all subsequent messages for that order. The held messages would have to be stored somewhere while the broken message is fixed. Then once the broken message is processed, all subsequent messages could be released. This will be quite tricky to implement
Many thanks for the response.
Yes it is tricky and maybe we are over engineering a problem in advance of experiencing it.
It sounds like fixing messages in flight is not something that is done per se? Some esbs like biztalk seem to offer this functionality out of the box.https://docs.microsoft.com/en-us/biztalk/esb-toolkit/collecting-exceptions-and-routing-messages-for-......but ActiveMQ and JMS say messages are immutable once they have been sent.
We have quite disparate systems that fire and forget, so data structures could change and we could get messages that the ESB deems as invalid and they need to go to a invalid message channel for manual inspection. If there is a required order to messages and some still are valid then we were wondering how to deal with maintaining a strict order.
Like you say building business and application logic to guard against things getting out of order sounds the way to go.
I'm just curious if there is any OOTB functionality we can make use of with talend and ActiveMQ.
Googling brings up concepts of ActiveMQ transactions, rollback, Message Groups to group messages together, Units of Work in Camel or other MQs like Websphere, strictOrderDispatchPolicies and blocking delivery, etc.
But I'm not sure if/how these concepts could be pulled together to help us?
Sorry, I may have misunderstood your requirement. When it comes to fixing messages in flight, I don't see how Biztalk can do that. If it is expecting .....
<data> <name>Richard Hall</name> <age>40</age> </data>
<data> <job>Community Manager</job> <location>London</location> </data>
..... it cannot possibly fix it automatically.
If you know the sort of issues you will get with your messages, then you can certainly build "fix in flight" into your routes with Talend. However I sense that something like this might be what you want https://help.talend.com/reader/1JiA7GVBIG_vkniH3yI8Mg/NqAMavU7YFHOqi6c9qkdbA
If you can define what you are looking for in a good message, you can use the example in the above link to help you send messages to an error queue or similar.
With regard to dealing with other messages from the same order, you will still have the issues I mentioned in my last post.
That's a pity AMQ messages are immutable
Thus, we decided to send wrong messages by email attachements to accurate user , then I can fix it and post it again in a directory --> cMessagingEndpoint --> processed again by the route.
Kafka could be a workaround, it offers easier replays for messages
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Learn how to use an API-First Approach to Modernize your Applications
Take a look at this technical overview video of Talend API Designer
Find out how to get started with APIs