That chevron button is the Associated Search feature. I don't know of a way to turn that off. But...in listbox Properties, General is a "More Search Settings" button. There you can limit what fields are included in Associated Search. That should address your performance issue.
I was able to go into the "More Search Settings" and use a "list of fields" but kept the list blank.
This eliminated the power of the chevron ">>" so it does not take any resources
It's NOT necessary to change the Default Search Option to "Associative Search". I did that earlier and it was throwing me off, since I lost the ability to do a fuzzy search.