Qlik Community

New to Qlik Sense

Discussion board where members can get started with Qlik Sense.

ziabobaz
Contributor II

stuck with incremental load

Hi,

I am trying to make incremental load from Google Tables.

The source have data for rolling 7 days (each day the earliest DATE is removed and NEW day is added).

I need to load only NEW and UPDATED rows.

I inserted the script from the HELP file (without really knowing what it means..), but the script returns the error:

Screenshot_62.jpg

I somehow need to tell Qlik not to Concatenate those records which were already loaded before.

Any suggestions?

Tags (1)
1 Solution

Accepted Solutions
ziabobaz
Contributor II

Re: stuck with incremental load

WHERE NOT Column2='cpc' AND NOT EXISTS(Date,Column1)

this one worked.

10 Replies

Re: stuck with incremental load

To being with it looks like your variable LastExecTime doesn't have any value. What LET or SET statement are you using to give it a value?


talk is cheap, supply exceeds demand
ziabobaz
Contributor II

Re: stuck with incremental load

Gysbert,

The problem is that I took this part of the script from the Help session and there is nothing there about LET or SET.

I don't get which date should I use to let QS know that i need to upload only those rows which were modified or new. I don't think it is about the date - there should be some kind of stamp or something telling which rows to check.

I am a front end user and not a programmer and this Help is not helpful at all:

Using QVD files for incremental load ‒ Qlik Sense

Maybe there is a simple example somewhere with step by step process?

tomasz_tru
Valued Contributor

Re: stuck with incremental load

How do you connect to the data source? Is there something like 'LIB CONNECT TO ...' before trying to run SQL query?

Tomasz

balabhaskarqlik
Honored Contributor

Re: stuck with incremental load

As per the Error Message there is no proper data connection

Where is the Database connection string, something like:

ODBC CONNECT TO connect-string OLEDB CONNECT TO connect-string CUSTOM CONNECT TO connect-string

LIB CONNECT TO connection


Try this:

Connect to data sources ‒ Qlik Sense

or

ODBC ‒ Qlik Sense

ziabobaz
Contributor II

Re: stuck with incremental load

Gents,

Right questions lead to right actions. I think the script above is not relevant to me.

My steps are:

1. Initial load from Google and saving as qvd:

STORE GoogleTable INTO [lib://QVDFolder/Коробка_LF.QVD] (qvd);


2. Load from qvd created in step 1

3. Load of data from Google Sheets

4. Saving (overwriting) INTO [lib://QVDFolder/Коробка_LF.QVD] (qvd);

Steps 2-4 are repeated every day.

So, how do I tell QS on step 3 that that it needs to update records which are both present in qvd and in Google and add new records which are present in Google and not present in qvd?

Highlighted
agigliotti
Honored Contributor II

Re: stuck with incremental load

to do an incremental load you need a primary key and a timestamp field using Exists function.

ziabobaz
Contributor II

Re: stuck with incremental load

Andrea, it does not tell me anything. Is there an example somewhere?

PS: let's make it simple - I don't want the records to be updated.

I need to concatenate only those raws where Date in the new database is not equal to the date from the OLD QVD (=add new records only). Is there way to do it?


GoogleTable:

LOAD

*

FROM [lib://QVDFolder/myQVD];

...

GoogleTable:

Concatenate

LOAD

*

FROM //googleurl


WHERE NOT Date //from googleurl// = Date //from qvd//;

ziabobaz
Contributor II

Re: stuck with incremental load

EXISTS is the key, I understand now.

Thank you, i will check the manual

ziabobaz
Contributor II

Re: stuck with incremental load

The script is working but is not loading the data as it is supposed to.

I think that this line is wrong: WHERE NOT Column2='cpc' AND EXISTS(Date);

    // Load from previously stored QVD

GoogleTable:

Load

*

FROM [lib://QVDFolder/my.QVD] (qvd);

    // Load from GoogleUrl

GoogleTable:

Concatenate

LOAD

   

Column1 as Date,

Column2 as Channel

FROM [GoogleURL]

WHERE NOT Column2='cpc' AND EXISTS(Date);

    // store QVD

STORE GoogleTable INTO [lib://QVDFolder/my.QVD] (qvd);

Community Browser