Incremental load

[#1]

Hi,

Anybody can help me on setting Incremental load on the below query.

I dont want to reload the old data every time I clcik on reload..

I want just the new data to be added.

Thanks

LOAD  Distinct

     CallID,

     OriginalParty,

FROM

The X directory receives new log files every 15 minutes and I want just to load the new files and add them.

[#2]

Hi,

I got an error  which is related to the stor CDR into CDR_Temp.QVD......I think that should be:

stor CDR_Temp_History into CDR_Temp.QVD

Under concatenate... the updated load comes from the same directory...the directory receives new log files simultaneously during the day..and for my case I want to avoid reloading the old data again I want to load only the new files that haven't been loaded in a previous load. so it doesn't take another 7 hours to finish the load.

One more thing how can I add a where statement in there. Like this:

Load ..

from...

where

    isnum(dateTimeOrigination) and not

(

(len(callingPartyNumber) = 7 and left(callingPartyNumber,3) = '810' )

or

(len(originalCalledPartyNumber) =7 and left(originalCalledPartyNumber,3) = '810')

or

(len(finalCalledPartyNumber) = 7 and right(finalCalledPartyNumber,3) = '810')

or

( len(originalCalledPartyNumber) = 0 and len(finalCalledPartyNumber) =0 )

or

( duration = 0)

or

(len(originalCalledPartyNumber)  < 10 and callingPartyNumber = 4)

);

Note that I am loading all this fields.

Thank you so much for your help and hope that will be helpfull.

[#3]

Rob,

The idea about this is to avoid long waiting load..

by using your method I am not sure if it is reloading again or not but it is still taking around 7 hours t finish the load.

[#4]

Hi,

I tried your method but something is going wrong specially that I have a where statement after the load..

I attached the file and a sample of the directory.

I would appreciate any help as i've been working on it for awhile.

Thanks,