Incremental load of logfiles - need help

[mjayachandran]

Hi all,

I'm trying to do an incremental load of log files. The unique key in the logfile is the 'Timestamp' field.

The log files format is as below:

Client_2014_08_10.log

Client_2014_08_11.log

Client_2014_08_12.log

Client_2014_08_13.log

Client_2014_08_14.log

and so on.

1. I want to store the log file into a month-wise qvd

2. I want to have the incremental load against it so that I don't waste time loading the same records again and again.

I am aware of doing incremental load of data from database but not sure how to do the same on logfiles.

Thanks for your help in advance !!

Regards,

Mahesh Jayachandran

[datanibbler]

Hi Mahesh,

the log_files are in most cases txt files. Have you checked that? They are quite easy to load using the "from document" wizard. I tried it with the QVS_logs and it does a good job.

Since you already know how to do an incremental load in general, I won't post that here. Just try.

HTH

Best regards,

DataNibbler

[mjayachandran]

Hi DataNibbler,

Thanks for posting . Can u elaborate a bit.
Sorry I'm a rookies to Qlikview

Thank you

[datanibbler]

Sure.

You save the log_file in a qvd, yes?

=> On the next day, you have to

- load that qvd into RAM

- concatenate a filtered LOAD from the log_files (just filter for the current day)

- Save it again, overwriting the old qvd.

That should be it, basically.

[mjayachandran]

Ok let me try and will let you know. Thank you.

[sudeepkm]

QlikView help has an explanation of incremental loading from log files like below.

Buffer (Incremental) Load * From LogFile.txt (ansi, txt, delimiter is '\t', embedded labels); // you need to define the location where the qvd is generated

other wise as DataNibbler mentioned you can loop through the files and can just load the file which has filetime() equals to today().

or if you are allow to move the log files to separate folder then you can use the execute command to move the log file immediately after loading data from it. in that case you will only have one log file in the log folder per day.