The calculations to generate the records may outweigh the time needed for the join. So your example may not be the right way to test this.
One scenario in which it would be faster to load the big table first is if the big table can be loaded with an optimised load from qvd. A join would cause a non-optimised load. The big table would suffer from this more than the small table.