topic JSON-tnormalize in Talend Studio

JSON-tnormalize

Anonymous — Mon, 04 Jul 2016 14:58:38 GMT

Hello,
I have a json file and normalizing some of the required field
my job flow is
tfileinputjson_1.>tmap_1.>tnormalize_1.>tnormalize_2.>tnormalize_2.>tnormalize_4.>tmap_2.>tuniquerow_1.>tfileoutputdelimited_1
in tmap_1 = adding fields having hardcoded value.
tmap_2 = applying some filtering on some json fields.
tnormalize_1 = field1 from json, tnormalize_2 = field2 from json, tnormalize_3 = field3 from json, tnormalize_4 = field4 from json
I have 3 more address fields , when I am adding 3 more tnormalize component, my job is working very slow/error.
I have increased java heap size , as more than 5 million rows in json.
Please suggest.
Thanks

Re: JSON-tnormalize

Anonymous — Mon, 04 Jul 2016 22:14:17 GMT

Because of the number of tNormalize components and the final tUniqueRow component it is quite clear you will overrun your memory.
tUniqueRow must collect all rows to take care the output is unique. What do you expect?
The solution must be you cluster your data reasonable and run your job only for such one data cluster. E.g. one country or one year or or...

Re: JSON-tnormalize

Anonymous — Thu, 14 Jul 2016 12:34:25 GMT

Thanks Jan, is there any way where we can normalize entire json meta data output using one component, right now I am using tnormalize for each column (value in comma separated arrays for ex and there are more than 10 columns
Below is for reference, after using jsonpath meta data using json file.
Example
ID     Name        Address                city
1        A
2        B

Re: JSON-tnormalize

Anonymous — Sun, 17 Jul 2016 17:23:26 GMT

What is your desired result? A value list over all?