topic Re: Batch JsonInput from Folder in Talend Studio

Batch JsonInput from Folder

Anonymous — Wed, 14 Feb 2018 10:56:12 GMT

Hi,

I am trying to extract some data in CSV format from around 40,000 Json files.

Assuming that all files contain the same fields, what would be the best/most effecient way to process these?

Thanks

Re: Batch JsonInput from Folder

TRF — Wed, 14 Feb 2018 11:03:36 GMT

Hi,

The genral design should look like this:

tFileList-->tFileInputJSON-->(tSomethingElseIfNeeded)-->tFileOutpoutDelimited

Click on the "iterate" link after tFileList then tick the option"Enable parallel execution" then enter the number of threads you want in parallel (maybe something like 4 or 5).

Re: Batch JsonInput from Folder

Anonymous — Wed, 14 Feb 2018 16:11:04 GMT

You are brilliant, thank you.
it is worth noting that in the filename path for the jsoninput you must specify the variable from tFileList_1:

((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))