Loading from Resident Tables Very Slow

[#1]

Hi -

I'm currently using SSIS with SQL Server to export a large number of raw text files for our hosted QlikView 10 Server to import and convert to QVDs.  We're looking at bring the server in-house, and in the process, I was hoping to eliminate the SSIS process altogether and import directly from QlikView.

In the short term, I'd like to load the SQL results into a resident temptable then use the current load script to grab the records from the temp table and then drop them into a QVD.  That works fine for our smaller queries, but for the ones with 1 million+ records, it takes substantially longer to load from the resident temp table than it does from the text file (even taking into account the time it takes to run the query).

Any thoughts as to why this might be?

Thanks

Pete

[Gysbert_Wassenaar · Accepted solution]

Try storing the table in an intermediate qvd after creating it and load from that qvd instead of using a resident load. That can be faster than using a resident load.

This blog post may be of interest too.

[#3]

That worked great - what was taking an hour with a resident load took two minutes using a temporary QVD instead.  It's odd that it isn't faster in RAM than reading it off disk, but it solves my issue regardless.

Thanks!