topic Re: Incremental Load using where clause in Qlik Automate

Incremental Load using where clause

Vikash2024 — Thu, 03 Apr 2025 08:26:18 GMT

Suppose I have a employee data. First I generate the QVD's. When i start to perform incremental load on QVD's using API's so i want only change data add in the QVD based on employee ID.
How to solve this problem?

Re: Incremental Load using where clause

PrashantSangle — Thu, 03 Apr 2025 08:51:27 GMT

Do we have any date field which tell us that there is change in this record??

if yes then use that field to do increment + update load.

regards,

Prashant Sangle

Re: Incremental Load using where clause

krishna20 — Thu, 03 Apr 2025 09:26:54 GMT

Hi @Vikash2024 ,

Try as like below.

QV_Table:

SQL SELECT Emp_ID, X, Y FROM DB_TABLE

WHERE ModificationTime >= #$(LastExecTime)#

AND ModificationTime < #$(ThisExecTime)#;

Concatenate LOAD Emp_ID, X, Y FROM File.QVD

WHERE NOT EXISTS(Emp_ID);

Inner Join SQL SELECT Emp_ID FROM DB_TABLE;

Re: Incremental Load using where clause

Vikash2024 — Thu, 03 Apr 2025 10:13:52 GMT

No there is no date field based on ID i need to capture the change and update according to that.

Re: Incremental Load using where clause

Vikash2024 — Thu, 03 Apr 2025 10:14:43 GMT

Thank you for the solution. But in the table i don't have date and time column.

Re: Incremental Load using where clause

PrashantSangle — Thu, 03 Apr 2025 10:38:20 GMT

If you don't have date time field then it is difficult to build incremental logic.

Do you have any indicator field which suggest that this column is updated.

Regards,

Prashant Sangle

Re: Incremental Load using where clause

krishna20 — Thu, 03 Apr 2025 10:49:16 GMT

Hi @Vikash2024 , Try this

IF isnull(QvdCreateTime('Incremental_Table.qvd')) then

Incremental_Table:
LOAD * INLINE [Emp_ID, Emp_NAME];

LET vMaxID = 0;

ELSE

Incremental_Table:
LOAD
Emp_ID,
Emp_NAME
from
IncrementalTable.qvd (qvd);

MAXID:
LOAD Max(Emp_ID) as maxID RESIDENT Incremental_Table;
LET vMaxID = peek('maxID', -1,'MAXID');
END IF

CONCATENATE (Incremental_Table) LOAD
Emp_ID,
Emp_NAME
from
SOURCE
WHERE Emp_ID > $(vMaxID);

STORE Incremental_Table into Incremental_Table.qvd (qvd);

Re: Incremental Load using where clause

Vikash2024 — Thu, 03 Apr 2025 13:11:10 GMT

Thank you for sharing the solution

Re: Incremental Load using where clause

Vikash2024 — Thu, 03 Apr 2025 13:11:41 GMT

Id column which can be use.

Re: Incremental Load using where clause

krishna20 — Fri, 04 Apr 2025 03:47:46 GMT

Try with your Emp_ID.