topic Re: Issue in incremental load in App Development

Issue in incremental load

iswarya — Wed, 29 Jan 2020 02:25:40 GMT

I store last 2 years of data in qvd and concatenate with daily refresh of 3 months data. But, after incremental load the values are not maching with the full load data. Any reason why the values not matching

Re: Issue in incremental load

Mauritz_SA — Wed, 29 Jan 2020 05:45:34 GMT

Hi @iswarya

It is very difficult to say why. Incremental loading is a critical component of various enterprise solutions and works well - it all just depends on the implementation. Why do you need to load the past three months' data and not only the new or updated records? Also, are you accounting for deletions if you have?

This article explains the basic concepts pretty well.

I am sure if you post your logic then someone on the Community will be able to help.

Regards,

Mauritz

Re: Issue in incremental load

HirisH_V7 — Wed, 29 Jan 2020 06:31:50 GMT

Could you please give much more info on which fields your'e considering for Incremental load i.e any key unique fields or date or any other.

Also let us know how many steps of increments your'e following : Insert, Update ,delete.

Better post some sample data which resembles your'e scenario, so that it will be easy for breaking it up.

Re: Issue in incremental load

jonathandienst — Wed, 29 Jan 2020 07:16:04 GMT

>>But, after incremental load the values are not maching with the full load data

The most likely problems are duplicates or gaps . If you are loading 3 months daily, how are you eliminating duplicates? That is what I would look at first.

Re: Issue in incremental load

iswarya — Wed, 29 Jan 2020 15:15:40 GMT

The data wont be updated or deleted. So I just concatenate the new data.

Re: Issue in incremental load

jonathandienst — Fri, 31 Jan 2020 06:20:18 GMT

If you are loading 30 days every day, then are you not creating duplicates of the last 29 days data?

Re: Issue in incremental load

Vegar — Fri, 31 Jan 2020 06:45:06 GMT

I'm in line with @jonathandienst , you're probably experiencing some overlap (or gaps).

I reccomend you to take look at this Qlik video as a start. Basic Incremental Load - Qlik applications