SCD TYPE 2 - Surrogate Key generation - Spark Big Data Job

shekchan — Sat, 16 Nov 2024 07:02:00 GMT

1. Create Context which saves MAX (surrogate_key) from Dimension table,

2. Create variable for Surrogate_Key ,

Variable (SK) = Numeric.sequence("COVER_CLASS_SK",(Integer)context.COVER_CLASS_SK + 1,1) ,

3. I understand running on multiple executors (Spark framework) the numeric sequence won't work,

4. Can someone please suggest with an example how to go about generating sequence numbers for dimension Surrogate keys?

Re: SCD TYPE 2 - Surrogate Key generation - Spark Big Data Job

vapukov — Mon, 17 Dec 2018 21:22:28 GMT

Hi Chandra,

just to keep it here as possible answer. Because it is just a surrogate key UUID could work good in this case.

it could be:

in case of primary key we do not need sequenced incremental, just must be unique

Vlad