topic Re: Newbie question regarding tMap columns limit in Talend Studio

Newbie question regarding tMap columns limit

Anonymous — Fri, 08 May 2015 09:26:26 GMT

I am evaluating Talend for a project which involves mapping data for tables that contain large number of columns, 500-2000. To work around tMap Java object size limitation, the recommendation I have seen is to break the processing into smaller sub jobs.
TMySQLInput_1 --> tMap_1 -->
TMySQLInput_2 --> tMap_2 -->
...
TMySQLInput_n --> tMap_n -->
What is a good way to join the output of all these sub-jobs together? I tried tHashOutput and tJoin, but it seems to be more work that it should be. Using pure SQL it would be a simple inner join statement. I suspect there is a better solution in Talend that I am not seeing.
Or even better, is there a better way to handle mapping for tables with 500-2000 columns?
Thank you.

Re: Newbie question regarding tMap columns limit

Anonymous — Fri, 08 May 2015 09:43:53 GMT

Hi,
Have you checked Talend "Dynamic Schema" feature which allows you to design schema with an unknown column structure (unknown name and number of columns).
Please take a look at document about:TalendHelpCenter:How to process changing data structure?

Best regards
Sabrina

Re: Newbie question regarding tMap columns limit

Anonymous — Fri, 08 May 2015 18:01:44 GMT

Looking at Dynamic Schema, I see it can be used to dynamically map source columns to destination columns, but I am not sure how it can help with what I am trying to do.
In this project, a table can have 500-2000 columns. I need to map the source table to a destination table with a predefined schema. The source table may share many column names and definition with the destination table, so had the source table been narrower (fewer columns), I can simply do an auto map for most columns, and manually specify the remainder.
Another solution that just occurred to me is to use multiple sub-jobs of tMySQLInput --> tMap --> tMySQLOutput, updating the output record with data from the subset of columns being process for the current sub-job. Is there a better solution that you can think of?