Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi im loading in over 10 mil lines from various sheets and performance is explanably poor. will storing the load into a qvd then loading that file improve performance?
Hi. QV will definitely load quicker a qvd file than any other source. However, to make it a qvd file, you need to load it from the database anyway, so the overall time might be the same. But you could use incremental loads and save a lot of time, especially if you have a unique key per record.
I think as giakoum , doing an incremental load its your best choice.
Ive seen that when you're transforming data, specifically with several millions of records, it is a good idea to save the intermediate tables into qvd's to free up space so that the others transformation processes have more space to work... and in my experience that does improves performance.
So.. I would say.. saving to qvd does makes a difference in certain tasks, but in your case it's a better choice to create an incremental load process.
Hi Samuel
The correct answer to your question is "Yes". The QVD files are meant to be a faster way to read data, but you have to keep in mind when you are loading the QVD file into you RAM memory you cannot add columns or make any calcultions because it will drop the benefit of loading faster, the only thing you are able to do without losing performance is to rename a column,
Wrong Examples:
Load
*,
1 as Column 2
From [youQVDfile.qvd](qvd);
//
Load
Date(dateColumn, 'YYYYMMDD') as DateColumn,
sum(valueColumn) as ValueColumn
From [youQVDfile.qvd](qvd)
Group By
dateColumn;
Correct Examples:
Load
*
From [youQVDfile.qvd](qvd);
//
Load
dateColumn as DateColumn,
valueColumn as ValueColumn
From [youQVDfile.qvd](qvd);
Hi,
Yes, it should be really faster to load from qvd than from the various sources. Especially since your qvd can be stored in a local file, while some of your sources are probably on network. Also, compared to csv for example, Qlikview doesn't have to read it all to reconstitute the cells.
Last, qvd takes usually less place on disk than other storages. Identical values are "grouped", let's say you have 1 million lines with an identical long string value, it will take 1 million times this size in csv, but not in qvd.
"will storing the load into a qvd then loading that file improve performance?"
Are you asking about the performance of chart objects? -- Or duration of script execution.
-Rob
Hi apologies for being away from the thread, im looking at improving the performance of the charts. theyre very slow.
then it wont make a difference.
Using a QVD to store data only improves the loading time of data and has nothing to do with the Chart performance.
If you want to improve the chart performance it is better to remove the complex calculations in the chart and do them in the script itself.
Hope that helps.
Or add more memory to your Server / Desktop.
Peter