Qlik Community

Ask a Question

QlikView App Dev

Discussion Board for collaboration related to QlikView App Development.

Announcements
Our next Qlik Insider session will cover new key capabilities. Join us August 11th REGISTER TODAY
cancel
Showing results for 
Search instead for 
Did you mean: 
mjayachandran
Creator II
Creator II

Incremental load of logfiles - need help

Hi all,

I'm trying to do an incremental load of log files. The unique key in the logfile is the 'Timestamp' field.

The log files format is as below:

Client_2014_08_10.log

Client_2014_08_11.log

Client_2014_08_12.log

Client_2014_08_13.log

Client_2014_08_14.log

and so on.

1. I want to store the log file into a month-wise qvd

2. I want to have the incremental load against it so that I don't waste time loading the same records again and again.

I am aware of doing incremental load of data from database but not sure how to do the same on logfiles.

Thanks for your help in advance !!

Regards,

Mahesh Jayachandran

5 Replies
datanibbler
Champion
Champion

Hi Mahesh,

the log_files are in most cases txt files. Have you checked that? They are quite easy to load using the "from document" wizard. I tried it with the QVS_logs and it does a good job.

Since you already know how to do an incremental load in general, I won't post that here. Just try.

HTH

Best regards,

DataNibbler

mjayachandran
Creator II
Creator II
Author

Hi DataNibbler,

Thanks for posting . Can u elaborate a bit.
Sorry I'm a rookies to Qlikview

Thank you

datanibbler
Champion
Champion


Sure.

You save the log_file in a qvd, yes?

=> On the next day, you have to

- load that qvd into RAM

- concatenate a filtered LOAD from the log_files (just filter for the current day)

- Save it again, overwriting the old qvd.

That should be it, basically.

mjayachandran
Creator II
Creator II
Author

Ok let me try and will let you know. Thank you.

sudeepkm
Specialist III
Specialist III

QlikView help has an explanation of incremental loading from log files like below.

Buffer (Incremental) Load * From LogFile.txt (ansi, txt, delimiter is '\t', embedded labels); // you need to define the location where the qvd is generated

other wise as DataNibbler mentioned you can loop through the files and can just load the file which has filetime() equals to today().

or if you are allow to move the log files to separate folder then you can use the execute command to move the log file immediately after loading data from it. in that case you will only have one log file in the log folder per day.