Incremental Load using where clause

[Vikash2024]

Suppose I have a employee data. First I generate the QVD's. When i start to perform incremental load on QVD's using API's so i want only change data add in the QVD based on employee ID.
How to solve this problem?

[PrashantSangle]

Do we have any date field which tell us that there is change in this record??

if yes then use that field to do increment + update load.

regards,

Prashant Sangle

[krishna20]

Hi @Vikash2024 ,

Try as like below.

QV_Table:

SQL SELECT Emp_ID, X, Y FROM DB_TABLE

WHERE ModificationTime >= #$(LastExecTime)#

AND ModificationTime < #$(ThisExecTime)#;

Concatenate LOAD Emp_ID, X, Y FROM File.QVD

WHERE NOT EXISTS(Emp_ID);

Inner Join SQL SELECT Emp_ID FROM DB_TABLE;

[Vikash2024]

No there is no date field based on ID i need to capture the change and update according to that.

[Vikash2024]

Thank you for the solution. But in the table i don't have date and time column.

[PrashantSangle]

If you don't have date time field then it is difficult to build incremental logic.

Do you have any indicator field which suggest that this column is updated.

Regards,

Prashant Sangle

[krishna20]

Hi @Vikash2024 , Try this

IF isnull(QvdCreateTime('Incremental_Table.qvd')) then

Incremental_Table:
LOAD * INLINE [Emp_ID, Emp_NAME];

LET vMaxID = 0;

ELSE

Incremental_Table:
LOAD
Emp_ID,
Emp_NAME
from
IncrementalTable.qvd (qvd);

MAXID:
LOAD Max(Emp_ID) as maxID RESIDENT Incremental_Table;
LET vMaxID = peek('maxID', -1,'MAXID');
END IF

CONCATENATE (Incremental_Table) LOAD
Emp_ID,
Emp_NAME
from
SOURCE
WHERE Emp_ID > $(vMaxID);

STORE Incremental_Table into Incremental_Table.qvd (qvd);

[Vikash2024]

Thank you for sharing the solution

[Vikash2024]

Id column which can be use.

[krishna20]

Try with your Emp_ID.