By "data extract" do you mean "running the reload script"?
I have at least one script that takes over an hour to run. If we had ten times the data volume (we're a relatively small company in the grand scheme of things), then it could easily take ten times as long. On the other hand, if we had ten times the data volume, we'd probably have ten times the processing power, and it would still take about an hour to extract.
I don't know much about SAP, but generally speaking, I would suggest that you only pull data from any given table or system once, and store it in a QVD. Make sure that you're reading using the right indexes, or whatever the fastest way to grab the source data should be. In some cases, you may want to add new indexes to the source tables to support faster data extraction.
Then the actual user applications would load only from QVDs, and not from SAP itself.
Then it's a matter of tuning all of your script in the applications. For large tables, you want to only load them once in the script, and to load using an optimized load. That puts some serious restrictions on the initial load. A technique I tend to use is this:
[Some Huge Table]:
LOAD * INLINE [
Some Field with the MOST Restrictive Values
Value 1
Value 2
Value 3
];
INNER JOIN ([Some Huge Table])
LOAD
"Some Field with the MOST Restrictive Values"
,other fields
FROM SomeHugeTable.qvd (QVD)
WHERE EXISTS("Some Field with the MOST Restrictive Values")
;
INNER JOIN ([Some Huge Table])
LOAD * INLINE [
Some Field with the Next Most Restrictive Values
Value A
Value B
Value C
];
INNER JOIN ([Some Huge Table])
"Unique Key Field"
,some expression as "Derived Field 1"
,some other expression as "Derived Field 2"
RESIDENT [Some Huge Table]
WHERE some final conditions
;
In my experience, this has usually been faster than putting the conditions and derived fields in the load of the QVD, as it allows for an optimized QVD load. The join at the end ends up being the slowest part, but it often seems to be faster to do the extra conditions and derived fields as a separate step. But definitely experiment, as what is fastest in one case isn't always fastest in another case. It may also differ between QlikView versions for all I know.
