Skip to main content
Announcements
See what Drew Clarke has to say about the Qlik Talend Cloud launch! READ THE BLOG
cancel
Showing results for 
Search instead for 
Did you mean: 
kdevarak
Contributor
Contributor

How do I read the Oracle table and split data into multiple files

Hi ,
I am looksng for a solution to read Oracle table with 10,000 records, the records need to be read 100 records at a time and converted into JSON and write to different files efficiently.
I am able to do the part of reading the table and writing to JSON, but unable to figure out how to split the data and write to different files. 
Also, when reading 10,000 records and writing JSON to a file the performance is very slow. 
Labels (3)
2 Replies
TRF
Creator III
Creator III

Hi,
Write 1st to tFileOutputDelimited and select "Spliy output in several files" option then indicate the desired number of rows.
After that, iterate over generated files then transform the content to JSON file (1 CSV = 1 JSON).
Regards,
TRF
kdevarak
Contributor
Contributor
Author

Thanks so much! I tried it and it worked! Another related question is ..how do I run a job in a loop until a variable in the global map is true? For example I want to read the db records and divide the number of records by a thousand to get the number of times to loop. Then loop the job multiple times  to read the Db and create the json . How can I loop the job multiple times?
Read Db and get count of records to create json
Store the rec count/1000 in var to get number of times the job should run to process 1000 recs
loop the job to process based on the var.
How do I achieve this?