topic Re: Incremental Log file load in QlikView

Incremental Log file load

Thu, 28 Aug 2014 18:34:55 GMT

Howdy,

I'm kind of confused about incremental load from log files...

Is it as easy as...

Buffer (Incremental)

Load *

From

O:\DavidC\3CX\*.txt

(txt, codepage is 1252, no labels, delimiter is ',', msq, filters(

Rotate(left)));

The question is, why is this not speeding up my reload times. I'm loading phone call log files, which grow daily. Shouldn't this script ONLY load the new files? Every day...my reaload time grows and grows as my number of log files.

-David

Re: Incremental Log file load

rwunderlich — Thu, 28 Aug 2014 18:52:05 GMT

Yes, it is that easy.

You will see "loading" messages for the "old" files, but it is only checking the filesize and not actually loading any data from the files -- it gets that data form the buffered QVD.

What I'm not sure about is the effect of the filters(). Can you try it without the filter as a timing test?

-Rob

Re: Incremental Log file load

Thu, 28 Aug 2014 18:53:28 GMT

Do you mean my "rotate(left" cross table part of the script?

Re: Incremental Log file load

rwunderlich — Thu, 28 Aug 2014 18:57:18 GMT

Yes, I meant remove the

filters(

Rotate(left));

I know it won't give you the correct result, but just as a test.

-Rob

Re: Incremental Log file load

Thu, 28 Aug 2014 19:22:05 GMT

OK, i'll update after the test.

What's the different in the script execution window between records with the "(buffered)" and the ones without?

-DC

Re: Incremental Log file load

rwunderlich — Thu, 28 Aug 2014 21:09:09 GMT

I think you see a "buffered" message at the beginning of the load, but other than that the progress window looks the same.

-Rob