topic Re: Incremental load in App Development

Incremental load

GOKULAKANNAN — Thu, 01 Sep 2022 17:15:35 GMT

I have to incremental load for below scenario,

It should insert and update

Id, place,count

1,chennai,10

1,chennai,11

1,chennai,10

2,madurai,10

2,madurai,11

2,madurai,10

In this i need to do increment load, where it should not remove duplicates but if any update is there it should do. Primary key can be created by concatenating all 3 columns, but i need all 6 records to be presented.

Re: Incremental load

deepanshuSh — Thu, 01 Sep 2022 17:44:02 GMT

Hi, since you have common data in all three fields with repetitions what you can do is use rowNo () function to create an addition column and then concatenate them together to create a composite key through which you can do the required incremental load. for the update part you need to direct it towards the Id as key if possible.

Re: Incremental load

rwunderlich — Fri, 02 Sep 2022 02:19:58 GMT

You'll need to create a new field to serve as the primary key.

Hash128(Id, place, count) as Key

-Rob