I never tested or even noticed it by the calc-times of a chart. But I noticed a similar behaviour within the script if a group by load is applied on a sorted table or on a not sorted table, like here described: Optimize Group By Performance.
I think it's technically quite the same and therefore I assume that the cause for the differences in the calc/run-times is also the same. As the reason I assume that Qlik needs just of few hops respectively processing-steps more by a disordered field as if it could read/process it in one continuous stream.
I can confirm I'm experiencing the same. I also did in previous 11.20.
With regards to the explanation, I'm sure Henric can provide one or get someone from R&D who can provide it.
Side note, in no small way this is why we have been moving as much as possible from JOINs to ApplyMap()s, taking more time to develop but not experiencing this difference.
If you just have sum(aggr(sum(LineTotal), OrderId, Product)) as the expression and X as the dimension for the pivot, then the calc times are reasonably close. This makes me think the order of the data is helping the pivot table draw the dimensions faster rather than helping with the actual calculations. It sort of makes sense since your order details pivot is 5M rows, the same size as the OrderDetail table and it helps when the data is already sorted in that order.
That said, I have seen calc times be significantly impacted by the order of fields in a table as well, which is more puzzling than the order of the records.