topic Re: Query_Incremental Load_Insert and Update in App Development

Query_Incremental Load_Insert and Update

rsdhavle — Sat, 16 Nov 2024 20:39:28 GMT

We have loaded data for 365 days in QVD. And have around 3L rows.

Now we have written incremental load as follows as we want to check last 60 days data and concatenate with existing QVD which has 1 year data

Load

ID,

Details,

Date

From QVD where Date > (vDate..last 60 days)

Concatenate

Load * from Existing.qvd where not exist (ID)

when we are doing this rows are getting reduced to around 75K. I am assuming its taking only last 60 days data. Ideally it should take for complete year and should check for 60 days for update.

Are we missing on anything?

Re: Query_Incremental Load_Insert and Update

arpitkharkia — Mon, 17 Jun 2019 08:53:34 GMT

By reading it i feel your ID may not be unique enough to use it as a condition. You may want to use a combination of your ID and Date as the unique ID.

Please try and let me know.

Re: Query_Incremental Load_Insert and Update

rsdhavle — Mon, 17 Jun 2019 08:58:30 GMT

Hi..yes we have ID as combination of Date and ID already..

Re: Query_Incremental Load_Insert and Update

arpitkharkia — Mon, 17 Jun 2019 09:01:47 GMT

try combining the 'Details' field also

Re: Query_Incremental Load_Insert and Update

arpitkharkia — Mon, 17 Jun 2019 09:11:37 GMT

or you can change your base table

Load * from Existing.qvd

concatenate

Load

ID,

Details,

Date

From QVD where Date > (vDate..last 60 days)