In general, use aggregation functions in expressions
Only() is an aggregation function, so that's what you probably should use here.
You can use set analysis only in aggregation functions, so that's one more reason to use one.
Now to your issue. I believe
Should filter your ID dimension, but maybe your color expression comes into play. How does it look like? Have you added a set expression also to your color expression?