Difficult to say without looking at the script. My suspicion would be ApplyMap() would be considerably faster in the final load but of course you have to take into account the step to generate the map.
In terms of CPU/RAM usage in the final application, they won't be any different as they are both using an integer key. The difference between the method will be in the load times.
I forgot to say... the performance issue is only in the QV Publisher Server side, relative to the loading process.
Prod server is 16 core CPU and 192 GB, and in the past we found that when the load process uses all the RAM available, the loading process fails.
QlikView version is v9.
Theorically, Option 1 (AutoNumberHash) could be more CPU consuming, while Option 2 (ApplyMap) could be more RAM consuming, assuming that all the Mapping table with up to 0.5 billion records needs to be loaded in memory.
Our test server is quite limited and the dummy files we got are not big enough to get confirmation of the above though.
We wish it were so simple but the client does not want to upgrade to QV10/11 for budget reasons, and we don't have access to Production at all. So our only option is do our best guess on what will have a minor impact on performance.
Then it is indeed a case of trial and measure! Personally, I love ApplyMap() - or even better, MAP USING (if possible). Other options are a LEFT JOIN with field dropping and renaming. For large data sets I have found this works extremely quickly.