Qlik Community

New to Qlik Sense

Discussion board where members can get started with Qlik Sense.

reivax31
Contributor III

Partial LOAD

Hi Qlikers,

I look at the different posts regarding partial Load with the function "Replace" or "Add". But in my case there is something I still can figure out.

Let’s say my script is like this:

DATA:

LOAD

*

FROM ... JAN 2016;

DATA:

LOAD

*

FROM ... FEB 2016;

DATA:

REPLACE LOAD

*

FROM ... THIS MONTH;

So every time it is loading all the files. But in fact I need only the data from the file “THIS MONTH” to be updated.

Because I have one big table called DATA, if I use REPLACE before LOAD for the file THIS MONTH, the only thing left will be the data from this month. But I want to keep all the data and replacing only the one for this month.

Is there a way to do so?

Thanks for your help

Tags (2)
13 Replies

Re: Partial LOAD

I guess replace load replaces the existing table with a new table. Do you have a different table name for the table that contains this month's data? Can you may be post your actual script?

MVP
MVP

Re: Partial LOAD

I think what you need is an incremental load, rather than a partial reload. Your incremental load would load the prior months from monthly qvd files. Although you are loading all the data, the qvd loads can be very fast compared to the original data source.

The load for the current month would continually update the qvd for the current month, so that when the month ends, it lands up as a history qvd. Something like:

Let vCurrentMonthFile = Date(Today(), 'MMM yyyy') & '.qvd';

DATA:     // replace with the correct load statement for the current period

LOAD *

FROM ......;

// replace <your file path> with the correct file path - replaces the current month file every day

STORE DATA into ['<your file path>$(vCurrentMonthFile)'];

// now load the history - make sure to not load the current month

For Each vFile in FileList('<your file path>\*.qvd')

  If SubField(vFile, '\', -1) <> vCurrentMonth Then

  DATA:

  LOAD * FROM [$(vFile)] (qvd);

  End If

Next

When the month ticks over, the previous month's qvd will contain the values for the month and you will start on a new qvd.

Logic will get you from a to b. Imagination will take you everywhere. - A Einstein
reivax31
Contributor III

Re: Partial LOAD

Hi,

I was already creating several QVD for each month. But still it takes a lot of time to load them all. So I found a way…

1st:

REPLACE ONLY LOAD

*

RESIDENT DATA

Where ImportDate<date(today(1));

2nd

LOAD

*

FROM THISMONTH.QVD

Where ImportDate>date(today(1));

Loading time is divided by 6:

Thanks for your help

reivax31
Contributor III

Re: Partial LOAD

Hi,

I think I made a mistake. I thought that it was possible to use RESIDENT DATA when I relaod my app. But it tels me that table is not found.

Is there a way to Call a table that already exist in my App?

Thanks for your help.

Re: Partial LOAD

During partial reload? Yes, of course you can.

Re: Partial LOAD

I think 1st is a table which doesn't exist in your application before the partial reload?

If that is true, you would need to do Add Only instead of Replace Only. But for every subsequent partial reload 1st will continue to exists. So to drop it each time, you can use an if statement:

If NoOfRow('1st') > 0 then

    

     DROP Table 1st

ENDIF

1st:

Add ONLY LOAD

*

RESIDENT DATA

Where ImportDate<date(today(1));

reivax31
Contributor III

Re: Partial LOAD

During Normal reload if I start with:

DATA:

REPLACE LOAD

*

RESIDENT DATA

where ImportDate<date(today(1));

I have an error message telling me that the table was not found.

reivax31
Contributor III

Re: Partial LOAD

I Guesse, During normal reload all the tables are drop at the beginning. How can I tell the system to never drop the tables?

Re: Partial LOAD

Is there any way you can share your script for normal reload and partial reload (essentially the whole script)?

Community Browser