Incremental load

[#1]

Hi,

Anybody can help me on this issue..

I've been working on it for over three weeks!!

I am loading phone log files"TXT" from a directory and the directory receives new log files every few minutes... and it is a hug directory"it takes over 8 hours to finish loading".

Is there anybody can help me in setting incremental load so I dont load all the files again.

It will be great if i can load to a QVD file and have the incremental load to there and then I can load from the qvd to my application.

Attached is the QVW and a sample of the directory.

Thanks

[#2]

Here we go.

I tried the one dot as well and it seems to have the same error.

Here is the attachement.

Thank you so much for spending the time helping me.

[swuehl]

There was just the trailing back slash missing.

I changed the logic, so you now specify the path including the file search string

SET vLogPath='.\log files test\cdr*';

maybe this makes it easier to maintain.

Attached the modified zip, should run out of the box.

[#4]

This sounds to be working fine.

How can I make sure it is not loading the files again ...I mean how can I test it?

Thxs

[swuehl · Accepted solution]

You'll see it in the log file or by looking at the script execution dialog.

Or by debugging (could be quite painful to go through hundred of files, so start with few).

You can add TRACE statements to the script to log more information at different point (please refer to the Help for that).

On large log volumes, you should notice an effect on load time (I even noticed when using the logs provided).

[#6]

Thank you so much for your grerat help.

I'll do that.

however, you are the best QV developer in here. I guess you have been working with QV for long ..long time.

[#7]

Hey Swuehl,

After loading the following script the data is storded in a qvd which is great, but when I add more Log files to the directory it doesn't add them to the QVD file..

Any advice will be great.

Thxs,

Badr

[hectorgarcia]

badr, you need to use the prefix "add"  in the load statement

Hector

[#9]

I tried Add Load *

but it didnt make any different

[swuehl]

I can't look into your problem in deep right now.

New files will only be read if the if condition of your script is fulfilled and records will only

be loaded when the WHERE clause is true.

So please double check both maybe by stepping through with the debugger.

I think I have tested my previous script with adding New files.

[sivarajs]

Hi,

Use can use Concatenation

The script will be like

Let Qvdpath=(path);

Temp:

load * from (Qvdpath);

concatenate

load * fom(LOG file which u are loading)

store Temp into $(Qvdpath);

Hope this helps....

Regards,

Sivaraj S