topic Re: QVD load in QlikView

QVD load

Tue, 23 Jul 2013 11:27:51 GMT

Hello

I have loaded data into my qlikview and i have above 70 mio rows. This takes quite long time to reload everytime, and to develop in, because i need to save all the time.

What is the best practice to do?

Is it A:

Data.qvw and store this into QVD.

QVDLoad.qvw -> Load data from QVDs and Design here?

Or is it B:

Data.qvw and store this into QVD

QVDLoad.qvw -> Load Data and make the nessecary scriptning

Design.qvw -> Binary load from QVDload.qvw and design in here.

Or what?

Re: QVD load

Tue, 23 Jul 2013 11:33:59 GMT

Data.qvw and store this into QVD.

QVDLoad.qvw -> Load data from QVDs (Only required period)and Design here..

Try to put less calculation in your UI.as much as do it in ETL time.

Re: QVD load

Joaquin_Lazaro — Tue, 23 Jul 2013 11:35:24 GMT

Hello Thomas Jensen

I suggest you:

1) DataHistoric.qvw ... create qvds with data from first day until 2012/12/31

2) DataAnual.qvw ... create qvds with data from 2013/01/01 until 2013/xx/31 (xx=previous month)

3) DataCurrent.qvw ... create qvds with data from 2013/zz/01 day until today (zz=previous month)

4) App.qvw ... load qvds* and contains all the charts and analysis insigths

Hope help you

Joaquín

Re: QVD load

brijesh1991 — Tue, 23 Jul 2013 11:35:58 GMT

Binary Load will be the best. ..

Also if you will make changes in Data Model file, then take partial load using add/replace function at scripting level. . .

Re: QVD load

kaushiknsolanki — Tue, 23 Jul 2013 11:36:57 GMT

HI,

Both the approaches are good.

But there are few things which you need to make sure to improve the performance.

1. DataModel.

2. Move all your script calculation to Qvd Generation, like any if statements or any Composite key generation.

3. Dont use if statements in charts.

and many more.

For more information have a look at the link below.

http://community.qlik.com/docs/DOC-1881

http://community.qlik.com/docs/DOC-1882

Regards,

Kaushik Solanki

Re: QVD load

Tue, 23 Jul 2013 11:38:27 GMT

I already did as much ETL calc as i can. And i need all period

Re: QVD load

Tue, 23 Jul 2013 11:39:42 GMT

Uhm. I see your point. But the fact is that i have some data, which is not dependent of a date. So i need ALL data.

Re: QVD load

Tue, 23 Jul 2013 11:43:26 GMT

Hello,

1. By this you mean (Have my datamodel in place) My DataModel is well considered, so i dont think i can gain much more performance or less rows here.

2. So all my scriptning in QVD. Thanks - That was also what I saw as the "best way"

3. Trying not to

I think im going with the binary load. It seems faster, and loads in few minutes, where the old way took 30 mins

Re: QVD load

Joaquin_Lazaro — Tue, 23 Jul 2013 11:45:44 GMT

Normally the fact table depends on date.

With qvd you'll also have all the data.

Try save all the data depending out of your document in qvds and load the app from qvds and mesaures/compare the duration of load script.

Re: QVD load

Tue, 23 Jul 2013 11:50:08 GMT

Yes! But some of my data is not date depended. Maybe 60mio of my rows is not date depended

Re: QVD load

Joaquin_Lazaro — Tue, 23 Jul 2013 12:02:41 GMT

OK.

Load the 60 mio records since today into qvds (only one time), and from tomorrow into new qvd (every day) ... this is incremental load

The app.qvw with chart load qvd very, very fast.

Re: QVD load

Tue, 23 Jul 2013 12:05:10 GMT

Well, the data changes everyday, so this is also not possible, and is always only showing todays data.

Re: QVD load

Joaquin_Lazaro — Tue, 23 Jul 2013 13:37:18 GMT

If you need retrieve 60 MIO records eveyr time (current status), you must wait.

But if you needn't retrieve 60 MIO every time, you can retrieve facts tables at 06:00am in a qvw and the other configuation tables (like Language-Texts) or other secondary tables (customer or items tables) in a second qvw at 07:00am.

Finally you'll need a third qvw with the insight charts that reloads from qvd at 08:00.

Other possibility is partial reload, execute parts of the script conditionally:

IF ispartialreload() THEN

........ here partial reload instructions

ENDIF

IF not ispartialreload() THEN

........ here complete reload instructions

ENDIF

... here partial & complete reload instructions

The LOAD must be preceded by ADD or REPLACE, please read the help file.

Re: QVD load

Wed, 24 Jul 2013 05:02:53 GMT

Thanks for the idea! i will keep that i mind when having incremental load data.