The restrictions in the General Filters are applied when extracting BEx Query results to QlikView.However if you have any restrictions for Default Values - as far as I recall, they will not be applied for the resulting data in QlikView.
I reckon extracting data from a BEx query using OLAP connector should utilize BWA.
Not sure what you refer to as SAP Query. Do you mean queries maintained in t-code SQ01? If so, please consider using the SAP Query Connector. The OLAP Connector works only with BEx Queries.
Finally, with regards to OLAP Connector performance. Did you try extracting the data from BEx queries using the BEx Connector (available since version 5.9)? It should have better performance than the OLAP Connector.
Anyhow, I would recommend you to read the SAP Connector User Manual and Release notes for the version you have installed.