Loading Data from SAP Clustered Tables

[avastani]

has anyone had good success loading data from clustered tables in SAP by applying filters to it?

I tried loading an 11 million record MSEG table but it runs out of memory in 2.5 hours.

I tried loading it with a filter and it runs for 4.5-5 hours and then abruptly shuts down even though RAM consumption is low.

Any suggestions appreciated.

[danielrozental]

Solution for cluster tables is, read the header table (MKPF), then read MSEG rows based on the MKPF returned rows like

WHERE DOCUMENT IN (DOC1, DOC2, DOC3, ..., DOCN)

[avastani]

that was my first approach and it would take over 6 hours and then automatically abort itself. any workaround from there.

[danielrozental]

Are you trying this incrementally? like reading just a few days of information at a time?

[avastani]

yes, but the first load is a full load.

i am trying from MSEG by first reading in the header from MKPF on an incremental basis and building the Doc List. then i try to read in MSEG using WHERE EXISTS() and that takes forever and fails. Then tried reloading without WHERE EXISTS() and it falls out of virtual memory.

I do a limited load of a 1000 records with no filter and that works just fine.

problem is to find the best course of action to build the first time and find a long term solution to keep on it. I am thinking I will have to put it in a FOR Loop and genereate either 1 month or 1 year at a time.

[danielrozental]

If you're working with large volumes you should definitely look into splitting the information in different QVD files.

You should always do a read using dates, for the first times just run the process for like a month data at a time and run it several times until you reach the current day, then do a last 3-7 days extraction.