Hi to everyone that reads.
I have a scripting process that download data from files, and once it made all the calculations, split the results into several files splited by year-month. the files has the data in such way, but it can contains data from older periods.
So I start processing them, and the firsts ones make it through very fast, but once I filled several periods (last 3 years) the process slow down. The slowing, is not because file size or type of data, is because the spliting has to rearrange the data in several files across all periods.
The problem I face, is that I can fill one period of data in 1 minute without no problem. If the file has only one period, the total time of the script is something like 4 minutes. But the splitting, take 1 minute for each period. So If I have to fill 36 periods (even If I only need to insert one record), It tooks 36 minutes. Take in mind that this has to be done for 600 files. And this is a test, because we have to do the same process for other things.
So I have the idea to paralellize it somehow, but I don't see a way to call for another qvw inside my script, or even stop it from working while I call another one with parameters, to try to run 5 or more at the same time. We have several cpu´s on the server, so that is no problem.
Is there a way to accomplish this???
