topic Re: How do I read the Oracle table and split data into multiple files in Talend Studio

How do I read the Oracle table and split data into multiple files

kdevarak — Fri, 24 Mar 2017 18:30:49 GMT

Hi ,
I am looksng for a solution to read Oracle table with 10,000 records, the records need to be read 100 records at a time and converted into JSON and write to different files efficiently.
I am able to do the part of reading the table and writing to JSON, but unable to figure out how to split the data and write to different files.
Also, when reading 10,000 records and writing JSON to a file the performance is very slow.

Re: How do I read the Oracle table and split data into multiple files

TRF — Fri, 24 Mar 2017 18:45:41 GMT

Hi,
Write 1st to tFileOutputDelimited and select "Spliy output in several files" option then indicate the desired number of rows.
After that, iterate over generated files then transform the content to JSON file (1 CSV = 1 JSON).
Regards,
TRF

Re: How do I read the Oracle table and split data into multiple files

kdevarak — Thu, 30 Mar 2017 04:08:36 GMT

Thanks so much! I tried it and it worked! Another related question is ..how do I run a job in a loop until a variable in the global map is true? For example I want to read the db records and divide the number of records by a thousand to get the number of times to loop. Then loop the job multiple times to read the Db and create the json . How can I loop the job multiple times?
Read Db and get count of records to create json
Store the rec count/1000 in var to get number of times the job should run to process 1000 recs
loop the job to process based on the var.
How do I achieve this?