Why some QVDs Incredibly Slow to Load

[mhaandreas1]

I have a dashboard where the load script is just a variation on:

Table1: LOAD * FROM [file1.qvd] (qvd);

Table2: LOAD * FROM [file2.qvd] (qvd);

Table3: LOAD * FROM [file3.qvd] (qvd);

Table4: LOAD * FROM [file4.qvd] (qvd);

No additional columns, no where clauses, no anything. Just LOAD *, and just QVDs. One of them, about 40 million rows, takes about 3 minutes to load the first 300/400 rows, then seconds to load the remaining millions. All the other QVDs load within seconds, as I would expect an optimised load to do. 

What type of things could be causing this? It's not anything to do with synthetic keys, because there aren't any. 

[mhaandreas1]

Some things that don't change anything including only loading in the one problematic table and/or changing the load order. 

[mwanders]

Hi Mhaandreas1,

I don't know the content of your QVD's but if the column names are the same then Qlik will concatenate them. In that case only the first QVD will be optimized and not the others.

[mhaandreas1]

Mwanders, 

Thanks for the idea, but the QVD isn't being concatenated with anything else. It also happens when I load it and no other tables into the dashboard. 

[mwanders]

Hi,

Hmmmm strange. Is your data the same in QVD in the first 40 million and the millions after that? Is your QVD a concatenated version of other tables? 

[mhaandreas1]

Yep, totally the same. There's nothing special about the first few hundred rows. 

[mwanders]

Hi,

Then I'm clueless without the specific QVD. If you can share this, i can test. But I don't think this is shareable 😉 

I hope somebody has some thoughts on this. Good luck!

Greetz!

[marcus_sommer]

The entries within the progess-window may mislead in some cases to what when really happens - for example by some heavier transformations like an aggregation with some group by fields for which Qlik needs to process the data before writing the results in the final table. This kind of processing isn't visible within the progress-window respectively may lead to a discontinuing refreshing of the display. By the most loadings you won't notice it because the display is delayed a bit but by the heavier one it may takes minutes before showing any further progress.

In your case that's not the case (at least not if it also happens by loading only a single table - by multiple ones it could of course have some impact because the load-order isn't meaningless even if it are all optimized qvd-loadings) but quite similar things could happens if a rather large file is accessed through a slow network/storage. In this case in may need minutes to transfer the data before Qlik could load them. I see this often if I don't work on the server else pull the files over the network to my local machine. Therefore take a look from where you pull this qvd - and at least for a testing copy it to a local storage and check the load-times again.

- Marcus