Incremental script reduced records from QVD

[prashantbaste]

Dear All

Hope you all are doing well.

I need your views/solutions for one of my issue -

Here it is incremental load happening for reports. An abnormal behavior that is observed is, on one day number of records are reduced in QVD (Which either should increase or should remained same before & after load completes).

Has anyone ever faced such issue earlier or does anyone know what kind of issue might happen in this case?

Thank you.

--

Regards,

Prashant P Baste

[robert_mika]

What's the conditions/ variables?

Is that the same day every time.

Share some data/screenshot or app.

[gajapathy74]

If there is any logic / filter (where condition) applied in your load script might cause an issue. Are you running in the same environment (folder)?

[prashantbaste]

Hi robert_mika‌ ,

Thanks for your valuable time.

Condition checked is dateadded > $(RStamp) or datemodified > $(RStamp) 

where -

dateadded is date when record is added into database

datemodified is date if existing record is updated.

Existing qvd data is appended to new/modified records by checking if keyfied does not exists in qvd data.

Is it right way of logic ?

this has happened first time. so can't say about everytime.

[prashantbaste]

Hello gajapathy74‌‌

Thank you for your valuable time.

Steps that are followed -

1. Getting max dateadded/ datemodified into variable RStamp.

2. extracting data from SQL Server where dateadded or datemodified from data is greater than  $(RStamp) 

3. if number of such records is greater than 0 then append qvd data to result-set else read only QVD data.

4. Store data into QVD.

is there any issue in  above steps?

[jonathandienst]

I see only one way that your logic would decrease the number of records - if there are multiple values of specific keyfields in your qvd, which is then modified in the source. One record would be added for the keyfield, and multiple records would be removed, leading to a reduction in the number of records. You could detect this by counting the number of distinct keyfields in the QVD before and after loading and comparing this to the total number of records.

[prashantbaste]

Hello jontydkpi‌‌

Actually there is issue with source table occurred   that there are only 60 records (of 3 distinct days)  remained & rest all records gets removed. Where as it should have at-least  60-70 thousands (or more than that) of records till now.

But then as per logic those records can be added or not based on their dateadded & datemodified.

But QVD should have data intact which is stored earlier. Please correct me if I am wrong somewhere or if I am missing anything.

[Anil_Babu_Samineni]

Can you post Increment Load Script. You might forget to drop the Parent Directory. We might help you if you share the Script

And, As i assume you've created loop for that.

[gireesh1216]

i am also getting same problem in my project...But in my project don't have primary key field....any one help me on this...

[gireesh1216]

And one more thing

How to find updated records and newly insert records in incremental load??