Newbie question regarding tMap columns limit

One Star

Newbie question regarding tMap columns limit

I am evaluating Talend for a project which involves mapping data for tables that contain large number of columns, 500-2000. To work around tMap Java object size limitation, the recommendation I have seen is to break the processing into smaller sub jobs.
TMySQLInput_1 --> tMap_1 --> 
TMySQLInput_2 --> tMap_2 -->
...
TMySQLInput_n --> tMap_n -->  
What is a good way to join the output of all these sub-jobs together? I tried tHashOutput and tJoin, but it seems to be more work that it should be. Using pure SQL it would be a simple inner join statement. I suspect there is a better solution in Talend that I am not seeing.
Or even better, is there a better way to handle mapping for tables with 500-2000 columns? 
Thank you.
Moderator

Re: Newbie question regarding tMap columns limit

Hi,
Have you checked Talend "Dynamic Schema" feature which allows you to design schema with an unknown column structure (unknown name and number of columns).
Please take a look at document about:TalendHelpCenter:How to process changing data structure?

Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Newbie question regarding tMap columns limit

Looking at Dynamic Schema, I see it can be used to dynamically map source columns to destination columns, but I am not sure how it can help with what I am trying to do.
In this project, a table can have 500-2000 columns. I need to map the source table to a destination table with a predefined schema. The source table may share many column names and definition with the destination table, so had the source table been narrower (fewer columns), I can simply do an auto map for most columns, and manually specify the remainder.
Another solution that just occurred to me is to use multiple sub-jobs of tMySQLInput --> tMap --> tMySQLOutput, updating the output record with data from the subset of columns being process for the current sub-job. Is there a better solution that you can think of?