topic Re: Incremental script reduced records from QVD in QlikView

Incremental script reduced records from QVD

prashantbaste — Wed, 25 Nov 2020 16:16:04 GMT

Dear All

Hope you all are doing well.

I need your views/solutions for one of my issue -

Here it is incremental load happening for reports. An abnormal behavior that is observed is, on one day number of records are reduced in QVD (Which either should increase or should remained same before & after load completes).

Has anyone ever faced such issue earlier or does anyone know what kind of issue might happen in this case?

Thank you.

Regards,

Prashant P Baste

Re: Incremental script reduced records from QVD

robert_mika — Tue, 25 Oct 2016 10:48:03 GMT

What's the conditions/ variables?

Is that the same day every time.

Share some data/screenshot or app.

Re: Incremental script reduced records from QVD

gajapathy74 — Tue, 25 Oct 2016 10:48:23 GMT

If there is any logic / filter (where condition) applied in your load script might cause an issue. Are you running in the same environment (folder)?

Re: Incremental script reduced records from QVD

prashantbaste — Tue, 25 Oct 2016 11:01:43 GMT

Hi robert_mika‌ ,

Thanks for your valuable time.

Condition checked is dateadded > $(RStamp) or datemodified > $(RStamp)

where -

dateadded is date when record is added into database

datemodified is date if existing record is updated.

Existing qvd data is appended to new/modified records by checking if keyfied does not exists in qvd data.

Is it right way of logic ?

this has happened first time. so can't say about everytime.

Re: Incremental script reduced records from QVD

prashantbaste — Tue, 25 Oct 2016 11:07:21 GMT

Hello gajapathy74‌‌

Thank you for your valuable time.

Steps that are followed -

1. Getting max dateadded/ datemodified into variable RStamp.

2. extracting data from SQL Server where dateadded or datemodified from data is greater than $(RStamp)

3. if number of such records is greater than 0 then append qvd data to result-set else read only QVD data.

4. Store data into QVD.

is there any issue in above steps?

Re: Incremental script reduced records from QVD

jonathandienst — Tue, 25 Oct 2016 11:29:21 GMT

I see only one way that your logic would decrease the number of records - if there are multiple values of specific keyfields in your qvd, which is then modified in the source. One record would be added for the keyfield, and multiple records would be removed, leading to a reduction in the number of records. You could detect this by counting the number of distinct keyfields in the QVD before and after loading and comparing this to the total number of records.

Re: Incremental script reduced records from QVD

prashantbaste — Tue, 25 Oct 2016 11:38:41 GMT

Hello jontydkpi‌‌

Actually there is issue with source table occurred that there are only 60 records (of 3 distinct days) remained & rest all records gets removed. Where as it should have at-least 60-70 thousands (or more than that) of records till now.

But then as per logic those records can be added or not based on their dateadded & datemodified.

But QVD should have data intact which is stored earlier. Please correct me if I am wrong somewhere or if I am missing anything.

Re: Incremental script reduced records from QVD

Anil_Babu_Samineni — Tue, 25 Oct 2016 11:42:20 GMT

Can you post Increment Load Script. You might forget to drop the Parent Directory. We might help you if you share the Script

And, As i assume you've created loop for that.

Re: Incremental script reduced records from QVD

gireesh1216 — Tue, 25 Oct 2016 13:10:45 GMT

i am also getting same problem in my project...But in my project don't have primary key field....any one help me on this...

Re: Incremental script reduced records from QVD

gireesh1216 — Tue, 25 Oct 2016 13:14:41 GMT

And one more thing

How to find updated records and newly insert records in incremental load??