One of our clients is facing terrible performance when extracting data from SAP (SQL-Connector) from test system. Select * from VBAP takes 10 mins for 50'000 records. In total, 7 mio records are available in VBAP (~ 24 hrs for 7 mio records...). All possible parameters in connector are set (JobClass, Buffer Percentage, PacketSize). We use message server host - but same performance problem via application server host. QV support is involved but no progress so far.
SAP core: 700
CPU: 8 cores
RAM: 10 GB
QVCONN: QvSAPConnector.dll 5, 5, 8698 SR3
Transports: E6DK900099 / E6DK900086
While loading data, only 2 CPUs are busy and 3 GB RAM are free.
SM50: Access to VBAP can be confirmed, but with an interval of 30 seconds only.
The same 30 seconds interval can be monitored on level hard drive access.
Most of the time, SAP is busy with the QV programs.
QlikView gets ~ 5'000 records every 30-50 seconds.
Initially, we didn't touch the connection string. Only after facing the performance issue, we started modifying. e.g. we increased BufferPercentage to 50 (default = 10) but then the job in SAP stopped with errors ("not enough buffer space available - decrease buffer percentage used to 10 or a lower level..."). Also playing around with other parameters didn't increase the performance. So, we went back to default.
We simulated an increment load - same slow performance but acceptable as we don't need to extract millions of rows (200'000 new / modified rows in VBAP per day). But before loading incremental, we need the initial (full) load. Looks like we slice the full load per month - for each table and month one separate extract. And then schedule 4 to 6 parallel jobs via publisher. that will take a couple of days to (fully) load the data from all the needed tables. And then set up incremental loads.