As far as I understand, a block chart is not spanned by two axis, it is composed of blocks of blocks, the larger / thicker blocks are 1. dimension, the second / thinner drawn blocks the second dimension. If you click on any segment, you kind of drill down on first dimension, another click will select / drill down on second dimension.
So it should be enough to first click on A, then on Yes to filter on the values you want.
It might work if you create a field that combines the 2 dimensions. This field would have the values ('A_Yes','A_No', 'B_Yes', 'B_No', etc.) Replace your second dimension with this new field. (I don't recommend it, but you can also try to use a calculated dimension.) You can control the background color manually to color all the Yes's and No's the same. This should give you the behavior you are looking for.