Query_Incremental Load_Insert and Update

[rsdhavle]

We have loaded data for 365 days in QVD. And have around 3L rows. 

Now we have written incremental load as follows as we want to check last 60 days data and concatenate with existing QVD which has 1 year data

Load

ID,

Details,

Date

From QVD where Date > (vDate..last 60 days)

Concatenate

Load * from Existing.qvd where not exist (ID)

when we are doing this rows are getting reduced to around 75K. I am assuming its taking only last 60 days data. Ideally it should take for complete year and should check for 60 days for update.

Are we missing on anything?

[arpitkharkia]

By reading it i feel your ID may not be unique enough to use it as a condition. You may want to use a combination of your ID and Date as the unique ID.

Please try and let me know.

[rsdhavle]

Hi..yes we have ID as combination of Date and ID already..

[arpitkharkia]

try combining the 'Details' field also

[arpitkharkia]

or you can change your base table

Load * from Existing.qvd

concatenate

Load

ID,

Details,

Date

From QVD where Date > (vDate..last 60 days)