Need to know if below requirement can be achieved

Six Stars

Need to know if below requirement can be achieved

Hi Experts,
I have used Talend Data Integration and I must say, it?s an wonderful tool. It has automated so many activities for us in recent past. But, am a newbie in Talend ESB.
I have a requirement to perform the below operations, and I just need to understand if Talend ESB /Talend MDM can help us in achieving it, so that I can start exploring more on Talend ESB/MDM:
I need to integrate 2 systems (A and B) where both the systems have exposed Webservices as well as Socket connectivity (FTP and SFTP) and they are unix based Enterprise systems. Now, I need to integrate those 2 systems which means there is a need for adapter in between. I require the adapter to perform the below operations:
1) Suppose in one of a directory in System A, I keep a text File containing 10000 records (one record in each column).
2) The adapter should FTP the File and perform the below operations
a) Put the File in a Queue (say for eg, Apache Active MQ)
b) Perform data transformations such that the File in un-batched/split into 50 records and each 50 records are converted/transformed into XML requests and stored in a database (say MySQL/Postgre since its intermediate data)
c) Every XML converted above is then transformed into a SOAP request that System B can understand
d) Send the SOAP request to the End Point URL that system B has exposed
e) Get back the response, push it into a Queue and store the response in an Response table of the Adapter database
f) Now, Rebatch the split up request into one File containing the success and failure records
g) FTP the success and failure files back to System A
3) Suppose there is an Exception anywhere then a mail has to be sent to mail IDs configured so that the user can rectify the problem and reflow the task from the place where exceptions occurred (This is the main requirement)
Can anyone please suggest me if it is possible using Talend ESB? We currently know that Activiti (a BPM Open Source tool) can perform this activity.
Please help..
Employee

Re: Need to know if below requirement can be achieved

Hi Arun
This can definitely be done with Talend ESB. The TESB is available via the Mediation perspective, so be sure to to use the correct perspective so you have the right set of tools on your palette. If you have not downloaded it yet, you will need the TOS for ESB edition which is available via our web site. Here are a few suggestions.
#1 Use the cFile component
#2a. route the cFile component to a cSlitter if you want to process individual records, or just convert the body to a string if you want to send as a single large message. I suggest the individual record approach.
#2b. The cSplitter above shoudl already have parsed the message. You can do batching with the cAggregator component at the commit stage to the database if you like. Use the camel-jdbc or camel-sql component for the database transaction.
#2c. Use the cCXF component for this step.
#2d. Are you using the ESB for exposing System B? No problem if you are. If you are and System B has a Java API / client you can represent that in cBean components if you like.
#2e. This is a simple route from cCXF to cJMS.
#2f. When you use the cSplitter component earlier, be sure to add a correlation id to the message. Now use the correlation id with a cAggregator.
#2g. Use the camel-ftp2 component.
#3. Use the ErrorHandler to catch exceptions at the route or context level. They get routed to a dead-letter-queue (this is just one particular exception management strategy). then from that dlq endpoint route as desired, in this case to a camel-smtp endpoint for email notification.
Six Stars

Re: Need to know if below requirement can be achieved

Hi Edward,
Thank you so much for your detailed explanation. I have started downloading the Talend Open Studio for ESB.
I have one query on your last suggestion on the exception scenario:
Please let me know, if it means that once user receives a mail that exception has occurred, he will have an URL support to Log in (or correct me if am wrong here) and correct the reason for the exception and restart the Flow from that point on-wards when the exception occurred. This is the main requirement from the customer.
Employee

Re: Need to know if below requirement can be achieved

You would have to program that part of the workflow. So for example, if you think of your large, end-to-end process as a series of shorter routes, say A, B, and C. The route fails for some reason at route B. The error handler is triggered and the message is sent to the dead letter queue. From there the dead letter queue might persist the message along with an appropriate retry command object in a datastore. The dead letter route would then send the email in question to the human user. The body of the email might well have a URL for the command object, or perhaps a derived URL for viewing the command object in a nice web application. The user gets the email and clicks on the url. The web application displays the error message and the proposed command object. Maybe the user needs to change some setting in the command object before resubmitting. They then click retry. The command object is then resubmitted at the sub-route point B. It is important to note in this context that it is assumed that the route is (and should be) basically stateless. Any relevant information is stored in the Message that is flowing through the route, and hence it is persisted.
Note that there is an extensive exception management policy available in Camel as well. See http://camel.apache.org/error-handler.html . So this scenario is really for business errors that are indeed recoverable. If you get an error such as a network error, you can set things like the number of retries via the OnException policy. Only after the configured number of retries are attempted and fail with the ErrorHandler actually be triggered.
Hope this helps.
Six Stars

Re: Need to know if below requirement can be achieved

Thank you Edward, wonderfully explained.