Question: Bigquery Incremental load

[rbfxkiryl]

Hi,

I tried to execute incremental load from BigQuery, but looks like it's not working.

Please advice.

Information which I used to implement it: Basic Incremental Load - Qlik Sense - YouTube

1 - Doesn't work:

// get last updated row

MaxKeyLoad:

Load MAX(time) as MaxID

FROM [lib://BI Qlik/BQ.qvd](qvd);

LET MaxID = Peek('MaxID',0,MaxKeyLoad);

// MaxID = 1530392398

BQ:

LOAD

    userId,

    clientId,

    date,

    time;

SELECT

    userId,

    clientId,

    date,

    time;

FROM `Streaming*`

WHERE time > '$(MaxID)';

Concatenate

LOAD * FROM [lib://BI Qlik/BQ.qvd](qvd);

STORE BQ into [lib://BI Qlik/BQ.qvd](qvd);

2 - Get data from BigQuery without incremental load is working ok.

// get last updated row

MaxKeyLoad:

Load MAX(time) as MaxID

FROM [lib://BI Qlik/BQ.qvd](qvd);

LET MaxID = Peek('MaxID',0,MaxKeyLoad);

// MaxID = 1530392398

BQ:

LOAD

    userId,

    clientId,

    date,

    time;

SELECT

    userId,

    clientId,

    date,

    time;

FROM `Streaming*`

WHERE time > 1530392398;

Concatenate

LOAD * FROM [lib://BI Qlik/BQ.qvd](qvd);

STORE BQ into [lib://BI Qlik/BQ.qvd](qvd);

[petter]

The Peek() needs the tablename in single quotation marks just as the fieldname.

Let MaxId = Peek('MaxID',0,'MaxKeyLoad');

[balabhaskarqlik]

LET MaxID = Peek('MaxID',0,'MaxKeyLoad');

[rbfxkiryl]

I tried it earlier, it doesn't work - the same result as without single quotation marks:

Loading data has been started and never finished.

"Started loading data

MaxKeyLoad << BQ Lines fetched: 1

Connected

"

[rbfxkiryl]

Who could assist with this issue?

Many thanks,
Kiryl

[jonathandienst · Accepted solution]

I don't think you need the quotes in the Where

WHERE time > $(MaxID);

[rbfxkiryl]

Many thanks!