Dynamic schema generation : DB migration

One Star

Dynamic schema generation : DB migration

I had to migrate from MySQL to SQLite and try and used Talend for that. Quite successful.
Fortunately in that project there were just 5 tables, so the easy way was to drop 5 tMysqlInputs and tSQLiteOutputs, connect them, and apply correct schema to each one, all works.
But in case if there are much more tables in DB, such approach looks rather boring..
What I've tried to do was dropping tMysqlTableList and tMysqlConnection for it. Then tMysqlTableList connect to tMysqlInput, which has TableName property set to be equal tMysqlTableList.CURRENT_TABLE value (actually, ((String)globalMap.get("tMysqlTableList_1_CURRENT_TABLE")) ) and Query property: "select * from "+((String)globalMap.get("tMysqlTableList_1_CURRENT_TABLE")). Thus, tMysqlTableList will iterate over database tables and tMysqlInput will read them and forward data to tSQLiteOutput (in my case).
But in this case both input and output have undefined schemas. The question is, if there is a way to get and set schema dynamically? Actually what is needed is to set schema name (it's possible to take them from repository). Or maybe there are some other ways to perform database migration using Talend?
Community Manager

Re: Dynamic schema generation : DB migration

Or maybe there are some other ways to perform database migration using Talend?

Yes, there is a advanced feature in commerial version Talend Integration Suite, with 'create job from templates' feature, you can migrate the whole db just by following the wizards, choose the source db and target db.
Best regards
Talend | Data Agility for Modern Business
One Star

Re: Dynamic schema generation : DB migration

Thank you for answering, Shong!
My aim now is to study some capabilities of Talend Open Studio, and what I see now is great, I must say! =) Great thanks to developers!
Concerning DB migration, I've looked through generated java code of the job, and it seems, that there is no some kind of "schema/metadata" object, bindable to input/output - SQL statements such as create table, select, insert are statically built into java code, so no dynamic schema change is available.. At least it seems to be so. Maybe will try later to generate xml for job (probably what TIS actually does).


Talend named a Leader.

Get your copy


Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables


How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration


Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now