Optimize Not Exists() Incremental Load

[micheledenardi]

Dear all,

I'm trying to speedup an SAP incremental extraction of a very large table, the script i'm using is this:

MSEG:
LOAD *,
	MANDT&'|'&MBLNR&'|'&MJAHR&'|'&ZEILE as [%Key MSEG Incremental];
SELECT *
FROM MSEG
	Where .... ;



Concatenate(MSEG)
Load *
From MSEG.qvd (qvd)
	where not Exists([%Key MSEG Incremental],MANDT&'|'&MBLNR&'|'&MJAHR&'|'&ZEILE);

Drop field [%Key MSEG Incremental];


Store MSEG into MSEG.qvd (qvd);

The SAP select is quite fast and in a couple of minutes all new record are extracted, the problem is to concatenate old values with the new ones.

To do this i'm using Not Exists() function by checking the concatenation of key fields of MSEG table.

This last step required more than 30 minutes... is there a way to speedup this check step ?

[JordyWegman · Accepted solution]

Hi Michele,

Try to already create the key in MSEG and don't build it in your not exists. This is probably the cause.

Jordy

Climber

[micheledenardi]

Thanks for the suggestion, it works.

The concatenation now is 10-15 times faster.