topic Loading Data from SAP Clustered Tables in QlikView

Loading Data from SAP Clustered Tables

avastani — Fri, 17 Feb 2012 17:14:01 GMT

has anyone had good success loading data from clustered tables in SAP by applying filters to it?

I tried loading an 11 million record MSEG table but it runs out of memory in 2.5 hours.

I tried loading it with a filter and it runs for 4.5-5 hours and then abruptly shuts down even though RAM consumption is low.

Any suggestions appreciated.

Loading Data from SAP Clustered Tables

danielrozental — Fri, 17 Feb 2012 17:30:42 GMT

Solution for cluster tables is, read the header table (MKPF), then read MSEG rows based on the MKPF returned rows like

WHERE DOCUMENT IN (DOC1, DOC2, DOC3, ..., DOCN)

Loading Data from SAP Clustered Tables

avastani — Fri, 17 Feb 2012 17:32:43 GMT

that was my first approach and it would take over 6 hours and then automatically abort itself. any workaround from there.

Loading Data from SAP Clustered Tables

danielrozental — Fri, 17 Feb 2012 17:37:52 GMT

Are you trying this incrementally? like reading just a few days of information at a time?

Loading Data from SAP Clustered Tables

avastani — Fri, 17 Feb 2012 17:43:34 GMT

yes, but the first load is a full load.

i am trying from MSEG by first reading in the header from MKPF on an incremental basis and building the Doc List. then i try to read in MSEG using WHERE EXISTS() and that takes forever and fails. Then tried reloading without WHERE EXISTS() and it falls out of virtual memory.

I do a limited load of a 1000 records with no filter and that works just fine.

problem is to find the best course of action to build the first time and find a long term solution to keep on it. I am thinking I will have to put it in a FOR Loop and genereate either 1 month or 1 year at a time.

Loading Data from SAP Clustered Tables

danielrozental — Fri, 17 Feb 2012 17:56:30 GMT

If you're working with large volumes you should definitely look into splitting the information in different QVD files.

You should always do a read using dates, for the first times just run the process for like a month data at a time and run it several times until you reach the current day, then do a last 3-7 days extraction.