I would rather look at this like the set identifier / bookmark in combination with the field modifiers define a set as a whole.
Is there an priority / order in applying the field modifier to the base set identifier? I don't think so, and I believe it shouldn't.
As it shouldn't make a difference to the aggregation result if you make a manual selection in field A before you make a selection in B compared to first B, then A.
What do you think? Have you any hints that there is a priority in applying filter?