topic Re: Parsing multiple input JSON files in Talend Studio

Parsing multiple input JSON files

Anonymous — Mon, 09 Mar 2015 04:13:36 GMT

Hello All,
We have a requirement to read data from multiple json files stored at a particular location and transform all data and finally insert it to a postgre db.
I am using Talend tFileInputJSON component to read json file, but i can read only 1 JSON file.
Since i am fairly new to Talend, looking for experts opinion here on how to tackle this situation.
Any support appriciated.
Thanks,
Manesh

Re: Parsing multiple input JSON files

Anonymous — Mon, 09 Mar 2015 06:24:57 GMT

Welcome to Talend!
follow the instructions to get your job done.
Your job design should be like this.
tFileList--iterate--tFileinputJson---tMap--youroutput.

Add tfileList component, set the directory path where all files stored.
Add file mask as per your file names default with "*.*".
Connect tFileList to tFileInputJson with "iterate" flow.
you will have " file name with path" in tFileList global variables. add this global variable to tFileInputJosn file path text box.

global variable which store the file name with path.

((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

Re: Parsing multiple input JSON files

Anonymous — Mon, 09 Mar 2015 07:15:41 GMT

Thanks for your help Umesh.. Multiple file parsing Worked perfectly !
One question, can tMap hold data from all the json files before inserting into db or do i need to insert one by one ?
Thanks,
Manesh

Re: Parsing multiple input JSON files

Anonymous — Mon, 09 Mar 2015 08:56:29 GMT

use tBufferoutput and input component to hold the result in buffer then insert this result in db.

Re: Parsing multiple input JSON files

_AnonymousUser — Mon, 09 Mar 2015 14:39:47 GMT

It's often easier to convert the JSON to a CSV flat file first then import that into tables.
There are some free converters like json-csv.com

Re: Parsing multiple input JSON files

Anonymous — Tue, 10 Mar 2015 02:46:59 GMT

Thanks Umesh and Holga for your inputs. I will try to follow it. Basically my JSON has few nested loops and i need to put some custom logic before inserting in database,so converting into csv may not help my purpose.

Re: Parsing multiple input JSON files

Anonymous — Tue, 10 Mar 2015 05:55:36 GMT

if you share input json file, then we can suggest better solution.

Re: Parsing multiple input JSON files

Anonymous — Tue, 10 Mar 2015 08:19:16 GMT

Thanks Umesh,

Below is my sample json file with very few fields,
{"prod_code":"2568945","default_sku_cd":"010101001","sku_list":,"color_details_list":,"size_details_list":,"length_details_list":,"price":"12.00",}

So here I want to achieve 2 things,
1st Job : read each file like above and Insert sku_list row by row into db, but before that i need to map fields like color_code in color_details_list array to get respective color name or if any field is null then replace by "-" before inserting record.
2nd Job : Insert level 1 fields (ex. prod_code,default_sku_cd,price etc) but before that loop through sku_list for few details (ex. sku_price,color_code,color_name) for sku_code matching with default_sku_cd and then add all these details as a single row.
Since i am fairly new to Talend, unable to decide what approach to take to find solution for above problems. Any directions, help is highly appreciated.