Attached is one way to do it without touching the data model since you don't want to do that. I used $Field as a hidden dimension in the chart, then picked the right field based on the value of $Field. I believe it could be easily extended to as many dimensions as you want to use.
Edit: There's no reason for the hidden dimension of $Field. That does nothing for us.