Trouble loading large qvd

[datanibbler]

Hi,

I have an app which is some day supposed to be a "live" counter for the shopfloor, reloading every 10min.

The trouble is, displaying the target_values - what should be done - is based on a qvd_file, which in turn is based on an Excel file which changes only once a week, so I can use the qvd no problem - but the issue is that it is just so HUGE. The qvd has 442 MB. Accessing it (the time it takes until the "script process window" starts showing any loaded records > 0) is over a minute - and another 30sec later, it is all done (including some LOADs from other tables as well ...

Is there any way I can amend that?

Thanks a lot!

Best regards,

DataNibbler

[marcus_sommer]

Hi DataNibbler,

is your qvd load an optimized load? Another possibility is to look are there fields with high cardinality data like RowNo-ID's or time-stamps and remove and/or split them.

- Marcus

[datanibbler]

Hi Marcus,

I have now split the qvd - there were figures from 2012 and 2013 in there, I now singled out a file for 2014 only which is, looky looky, less than half the size.

That's still 180MB, however. It still takes about 30sec - considerably less than before - until the records start being loaded - locally, the server would of course be faster. I could take this further and make it a lot slimmer still by filtering that huge file for only the current day and every day generating a qvd file with the current day's data that I can

subsequently use for my ticker.

The question is only how to do this - should I insert the current date-stamp into the file_name, then I would probably have an issue deleting the files automatically. If I don't, I have to find a way to find out if there is already a file with today's date or not - I would like to have it all in one app:

- The code generating a day-specific qvd (which would run only once a day) and

- The code using this qvd to display the ticker data.

Can you help me find a way to do this? Is there something I can use to find out if a file exists? - Ah, I guess I can use FILETIME(), no?

Best regards,

DataNibbler

[marcus_sommer]

Hi DataNibbler,

yes you could use filetime() to check if a file is exists.

Perhaps you could split high cardinality fields into two fields - the amount of RAM or the filesize will be decrease noticeable. And then is this load an optimized load?

- Marcus

[Colin-Albert]

Is the initial pause due to "Save before Reload" being enabled?

[datanibbler]

Hi,

regard this as solved.

I have split out a file for this year and then I've taken it one step further and generated a qvd just for the current day - now I have a code to check whether there already is a daily_file - if there isn't (upon the first reload of the day), it is created, afterwards the current one is just used.

That got the qvd (well, the one in use, I have kept the others, too) down to about 1MB, the qvw has gone down by about 75% of its size and it now runs in about 1/3 of the time 😉

Thanks a lot!

Best regards,

DataNibbler