topic Re: Incremental Load in QlikView

Incremental Load

ferha_jafri — Thu, 06 Jul 2017 04:37:09 GMT

Hello Experts,

I need help in understanding the data which is excluding from qlikview after incremental load. I have a data in which we are using Output key as a key which we are using in incremental load to exclude duplicates and update the data . I understand that my data it is not unique due to which data get reduced after incremental load. My question is which record Qlikview is including while removing the duplicates. I am attaching the raw data and data after incremental load. Please help me in understanding the functionality of qlikview.

New is after increment and Increment has the raw data.

Thanks in advance

Ferha Sharma

Re: Incremental Load

Muthu — Thu, 06 Jul 2017 05:25:18 GMT

Please go through this link. You will get better understanding on Incremental Load

https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

Re: Incremental Load

ferha_jafri — Thu, 06 Jul 2017 05:53:47 GMT

Hello Muthu,

I understand incremental load my issue is that I have to answer the user that which records are not coming and why? after incremental load.

Re: Incremental Load

BalaBhaskar_Qlik — Thu, 06 Jul 2017 06:48:47 GMT

May be this:

How do you collect the data in your application?

For example, do you use an incremental load approach or a partial load?

You can think about creating a flag added to your table indicating the new records or an additional table linked to the records key field with that flag, you just need to ensure that the field / table is reset before the next load.

Or create just an additional field with the load time stamp or an load incremental identifier and filter the records for the max value in your front end to show latest records.

First load qvd

then load from excel only new record (=record not loaded from qvd) and add a flag for these records

store in qvd (without the flag).

modify the script like this:

LOAD Key, ..., 'OLD' as Status FROM Historic.qvd (qvd);

LOAD Key,... 'New' as Status FROM ...

WHERE NOT EXISTS(Key);

Re: Incremental Load

ferha_jafri — Thu, 06 Jul 2017 06:59:56 GMT

Bala ,

I could not understand what you suggested , as for your information I am using incremental load . My issue is when you check the attachment of the raw data and after incremental you will data is missing corresponding to Output Key I want to know on what basis it is getting those records or why it is not taking let records.

Thank you.

Re: Incremental Load

BalaBhaskar_Qlik — Thu, 06 Jul 2017 09:50:31 GMT

Add individual Flags like: Old, New as Status of records.

Then you'll came to know what are all the New, Old records & why they got rejected.

Re: Incremental Load

ferha_jafri — Thu, 06 Jul 2017 10:11:41 GMT

It will take the value which will come first , checked from the source data.