Qlik Community

Ask a Question

QlikView App Development

Discussion Board for collaboration related to QlikView App Development.

Welcome to our newly redesigned Qlik Community! Read our blog to learn about all the new updates: READ BLOG and REPORTED ISSUES
Showing results for 
Search instead for 
Did you mean: 

Challenge. Loading a bunch of excel files in an efficient way

Hi guys,

I have a situation like the following :

  • There's an automated process which creates excel files from first record to last record in a range of 500k records (generating new records every day) like this :
    • Excel_1_500000
    • Excel_500001_1000000
    • Excel_1000001_150000
    • ...

If I do this :

MyJoinedTable :






(ooxml, embedded labels, table is Sheet1);

I get a new table which loads all the records available in the excels, and it's fine.

But right now my load script takes more than an hour, and it keeps increasing every day.

What would you do to improve it?

Regards, Marcel.

5 Replies

See attached a sample app and data used

Basically for the first load (based on a flag) load the file names and data into a qvds

Subsequent loads

- get list of files names

- load old list of file names from qvd

- create another table with latest filenames only

- load data from qvd

- loop through new file names and load which gets autoconcatenated

- drop unnecessary tables (havent done this in the app to prove it works)

- save qvds

hope it helps


did it work?

Contributor III
Contributor III

Can you get the data from the source that creates the excel files?  If so, you get set up an incremental load from the source.



Thank you Dilip, it was a very complete advice.

Finally what I'm doing is your for next loop and storing each table in a single QVD, it reduced the reload time from 1 hour to 10 minutes.

After the first reload, I check the filesize() of the excel to decide if I have to store it again as QVD or not.

Regards, Marcel.


Good to hear

Thank You

Dilip Ranjith

Sent via mobile