I am evaluating Talend for a project which involves mapping data for tables that contain large number of columns, 500-2000. To work around tMap Java object size limitation, the recommendation I have seen is to break the processing into smaller sub jobs. TMySQLInput_1 --> tMap_1 --> TMySQLInput_2 --> tMap_2 --> ... TMySQLInput_n --> tMap_n --> What is a good way to join the output of all these sub-jobs together? I tried tHashOutput and tJoin, but it seems to be more work that it should be. Using pure SQL it would be a simple inner join statement. I suspect there is a better solution in Talend that I am not seeing. Or even better, is there a better way to handle mapping for tables with 500-2000 columns? Thank you.
Looking at Dynamic Schema, I see it can be used to dynamically map source columns to destination columns, but I am not sure how it can help with what I am trying to do. In this project, a table can have 500-2000 columns. I need to map the source table to a destination table with a predefined schema. The source table may share many column names and definition with the destination table, so had the source table been narrower (fewer columns), I can simply do an auto map for most columns, and manually specify the remainder. Another solution that just occurred to me is to use multiple sub-jobs of tMySQLInput --> tMap --> tMySQLOutput, updating the output record with data from the subset of columns being process for the current sub-job. Is there a better solution that you can think of?