How do I set conditional colors for a chart based on Y-value?

[#1]

I am charting survey results (see image below). I want to maintain the color scheme below (green for best to red for worst). The issue is that some results sets are so small that there could be only 1 response, and if that is 'Never' then it appears as Green since that's the first color in my scheme.

How can I use conditional formatting so that 'Always' is set to Green, 'Never' is set to Red, etc?

All the examples I've seen here for conditional colors seem to be based on the values (count or sum).

Thanks

[#2]

Thanks Karla and Kumar!