Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi guys,
I have a situation like the following :
If I do this :
MyJoinedTable :
LOAD
fieldA,
fieldB
FROM
[excel_*.xlsx]
(ooxml, embedded labels, table is Sheet1);
I get a new table which loads all the records available in the excels, and it's fine.
But right now my load script takes more than an hour, and it keeps increasing every day.
What would you do to improve it?
Regards, Marcel.
See attached a sample app and data used
Basically for the first load (based on a flag) load the file names and data into a qvds
Subsequent loads
- get list of files names
- load old list of file names from qvd
- create another table with latest filenames only
- load data from qvd
- loop through new file names and load which gets autoconcatenated
- drop unnecessary tables (havent done this in the app to prove it works)
- save qvds
hope it helps
did it work?
Can you get the data from the source that creates the excel files? If so, you get set up an incremental load from the source.
Cathy
Thank you Dilip, it was a very complete advice.
Finally what I'm doing is your for next loop and storing each table in a single QVD, it reduced the reload time from 1 hour to 10 minutes.
After the first reload, I check the filesize() of the excel to decide if I have to store it again as QVD or not.
Regards, Marcel.
Good to hear
Thank You
Dilip Ranjith
Sent via mobile