Optimising load of csv files

[coolguy123]

ola!

 My system currently exports csv files automatically daily to a folder stucture

——-like this 

data (main folder)

old data (this folder is within main folder + yesterday data as csv files)

within old data I have more folders dated everyday  e.g. 2021-02-01 etc

———

so data (this is a folder) > old data (this is also a folder and has more folder in this that are dated) + yesterday csv 

the the next day yesterday csv get put into a folder into old data automatically and new csv are downloaded 

——

what is the best way to load these I have thousand of files and it takes a long time currently? Maybe I can convert these to QVDs?

I have a sub routine that looks at all of the folder and loads from the start everyday but this takes a long time I need to optimise this? For faster load speed?

something like for each vfolder in dirlist ( c:/data/old data/*)

I then concatenate yesterday csv onto this as there is no folder for it (c:/data/old data/files.csv)

[tm_burgers]

What about loading them all through your subroutine once; and then just Concat the new daily file each day to a QVD. 

You could define the file path and the file name with Variables for today. 

Basic Structure:

1. Run your non-optimized subroutine one time and store the resulting table as a QVD file (you can even do this is a separate app that you create just for running this process)
2. In your production app, load the QVD 
3. Define the file path for "Todays" file(s)
4. concat new data to historical data QVD table
5. Store new table as a QVD (either overwriting existing, or a new one each day with a daily variable, but then you need to adjust Step 2 to look for the correct "newest" file).