Start Schema on big data set

[sujit_nath]

I'm working on a large data set with csv file with 7-10 gb size. Now I need to combine 2(or more) files on a key field that is alphanumeric. How should I acheive this in the most optimized way?

[marcus_sommer · Accepted solution]

To associate the tables within a qvw you could use: autonumber(YourKeyField) as KEY. But if you need this key already before, for example for incremental approaches autonumber() might not be suitable anymore.

An alternatively could be to replace the alpha-numeric key with a numeric ones, for example by summing/multiplying ID fields instead of string-concatenate them, for example one of my most important keys look like:

([StoreID] * pow(10, 4)) + ([OrderID]) * pow(10, 8)) + [PosID] as [UniqueKey]

Beside this the data-size in Qlik might be significantely smaller as the rawdata especially if there are only a rather low cardinality of the field-values and if not it might be worth to look if it could be optimized in that way.

- Marcus