Incremental Loading and Autonumberhash256 for a very large application

[#1]

Hello,

I have a problem - a BIG one.

I have a  dashboard which is integrating a lot of disparate sources and  the link table I have has 6 keys. Each of them is a composite key and I had used Autonumberhash256 to minimize the size.

Now the problem is

1. we need to incrementally load

2. The link table is over 120 mn records.

I am aware now that autonumberhash256 will not work during incremental loading after I saw hash collisions in the key. So same key generated for two different periods where the original values are different.

I thought of using the hash256 in the qvd generator level for incremental loading and using autonumberhash256 in the model. But I am not too sure it will perform well.

I do not know of a way to keep the autnumbering persistent across different runs (incremental).

I cant move this to ETL tools so have to resolve this is QV.

Any bright ideas?

[marcus_sommer]

It seems that our data-structure within the ERP system is different from yours. But this isn't important - essential is if it's possible to create unique numeric values by the combination from different numeric ID's.

And don't forget my last sentence in my first answer to check if other approaches within the datamodel could suit your requirements better then a link-table approach.

- Marcus

[herbert_beck]

Would you please share the key values and the hash-function that caused the hash collision.

Kind regards

Herbert