One Star

ETL process testing automation

Hi all
I've had to create some tests for Talend jobs recently and I wonder whether the chosen way was right or not.
The simplest tested process took data from MsSql database, transformed them via tMap and then loaded them into MySql table. In order to test it, I?ve had to switch its input to delimited file, so the job started to look like this:
tMSSqlInput --------tUnite------tMap----------tMySqlOutput

File and table have the same schema, and tMSSqlInput select query looks like this
"SELECT * FROM myTable" + (context.is_test ? " WHERE 2 = 1" : "")
So when the job is launched with the test context, no data flows from MsSql to tUnite. The same is right for tFileInputDelimited and non-test context.
Next steps include feeding test cases files to the target job, dumping resulting MySql table and asserting resulted file with dump etalon using tFileCompare, tAssert and tAssertCatcher.
The main limitations I?ve encountered were these:
1. There is no way to create some template for such tests because of static table schema;
2. Each tested job now holds one or more components for mocking input;
I?m relatively new in TOS, and hope to hear opinions from more experienced people.
And one more thing: have anyone tried Talend Testing Platform in action? It looks pretty useful from the promo page, but I failed to find more information about its possibilities. Which edition of Talend Integration Suite (Team, Professional or Enterprise) contains Talend Life Cycle Management Products, or are they stand-alone components?
One Star

Re: ETL process testing automation

Welcome to Talend Community!
According to your requirement, I know that you need two features which are only supported by Talend Integration Suite(Commercial Version).
These two feature are joblet and dynamic schema.
Sorry for this. Because you are using TOS now.
Besides, you may find more info here about Talend Life Cycle Management Products.
One Star

Re: ETL process testing automation

Hi, Pedro
Thank you for your reply!
I?ve re read about dynamic schema and joblets and they seem just right for the task. That is why my employees are now seriously thinking about buying commercial version of the Talend, and I?m trying to find out, which version we exactly need. According to the table here, joblets and dynamic schema are available in all versions of Talend Enterprise Data Integration, but what about Talend Life Cycle Management Products? I haven?t found them in the comparison table.
Kind regards,
One Star

Re: ETL process testing automation

Hi Shentarelle
You can subscribe Talend Enterprise Data Integration from Talend Website and get more info from Sales Team.
They will show several professional documents.