topic Re: Scan file system in optimized way! in QlikView

Scan file system in optimized way!

qlikviewforum — Thu, 21 Nov 2013 13:29:25 GMT

Hi All,

Is there is any optimized way to scan the file system other than using the below code which generates very large log file. Please help!

  
// list all QV related files on disk 

sub DoDir (Root) 
for each Ext in 'qvw', 'qva', 'qvo', 'qvs' 
for each File in filelist (Root&' \*.' &Ext) 
Load '$(File)' as Name, FileSize( '$(File)' ) as 
Size, FileTime( '$(File)' ) as FileTime 
autogenerate 1; 
next File 
next Ext 
for each Dir in dirlist (Root&' \*' ) 
call DoDir (Dir) 
next Dir 
end sub

Re: Scan file system in optimized way!

Bill_Britt — Thu, 21 Nov 2013 14:12:08 GMT

Hi,

I would once you had everything working like it should, turn off the logs.

Bill

Re: Scan file system in optimized way!

qlikviewforum — Thu, 21 Nov 2013 14:37:58 GMT

Hi Bill,

Thank you!

We do need logs to be saved for our investigation. So I would like to optimize it by doing some changes in the code and let it saves the log files in the normal way. If I can optimize it then that could be the best way to do it instead of thinking about how to off the logfile.

Though it will not be accepted, please let me know how to turn off the logs if everything it working file. I am not sure how to do that.

Thanks

Re: Scan file system in optimized way!

DavidFoster1 — Thu, 21 Nov 2013 15:44:40 GMT

isnt the end sub on the wrong line? Shouldn't it be on line 12?

Re: Scan file system in optimized way!

DavidFoster1 — Thu, 21 Nov 2013 15:46:00 GMT

document properties -> general tab -> generate logfile flag

Re: Scan file system in optimized way!

qlikviewforum — Fri, 22 Nov 2013 02:59:17 GMT

No, it will be there only.

Re: Scan file system in optimized way!

qlikviewforum — Fri, 22 Nov 2013 03:00:32 GMT

I know this properties but is there is anyway where we can generate the log file only when the application fails. This is what my question was!

Re: Scan file system in optimized way!

DavidFoster1 — Fri, 22 Nov 2013 07:07:04 GMT

// list all QV related files on disk
sub DoDir (Root)
for each Ext in 'qvw', 'qva', 'qvo', 'qvs'
for each File in filelist (Root&' \*.' &Ext)
Load '$(File)' as Name, FileSize( '$(File)' ) as
Size, FileTime( '$(File)' ) as FileTime
autogenerate 1;
next File
next Ext
end sub
// Loop through directory structure calling DoDir procedure
for each Dir in dirlist (Root&' \*' )
call DoDir (Dir)
next Dir

There is no way to avoid the logging as it is generated at the same time as the action. It is the individual load statements for each file that will be bloating the log file.

Re: Scan file system in optimized way!

qlikviewforum — Fri, 22 Nov 2013 07:50:31 GMT

Hi David,

Thank you for your response.

Is there is any way to restructure the code in better way to avoid logging and the time it takes to refresh. Some how I need to restructure this code to avoid logging. If not possible to restructure the same piece of code we are ready to switch if the new code doesn't takes much time and have less logging.

I think oneway of optimizing this code is to scan only those files which we are looking for instead of scanning each and every file. For example if we looking for xls file in any of the directory code should be in a such a way that it scans only the xls file instead of scanning csv,jpg and etc...I agree it looks in only for xls file it creates the unnesscary logging if the directory contains other file format like csv,jpg and etc...

Please let me know if you have another alternative for this.

Thanks,

Re: Scan file system in optimized way!

DavidFoster1 — Fri, 22 Nov 2013 08:23:01 GMT

There is no way to simplify the code any further. I think to meet your requirements you will need to go back to looking at something like DIRUSE to extract the data and then load it into QlikView for analysis.

Re: Scan file system in optimized way!

qlikviewforum — Fri, 22 Nov 2013 09:37:42 GMT

Hi David,

That I have already been trying but still no luck to read the data from unstructured file format. Below the thread which I have created for that. It would be great if you can provide some solution there.

Reading line by line to get the size of each file with full path

Thanks,