topic filelist() problem with large data amounts in QlikView

filelist() problem with large data amounts

saxjonas — Fri, 03 Dec 2010 13:07:09 GMT

Hi everyone!

I've stumbled across a problem with the filelist() function.

What I'm doing is reading the filenames of files in a certain folder. This specific folder has over 3,000,000 files. It is running fine with around 600,000 files but gets stuck after about 1,000,000 - 1,500,000 files. This occurs even though there is still memory free.

My thoughts are that it might be a limitation on the underlying function. I'm guessing filelist uses filesystemobject or something like that.

Any ideas?

filelist() problem with large data amounts

Fri, 03 Dec 2010 13:37:19 GMT

Hi,

It is not "best practice" to store that many files in same folder, regardless of operating system

-----

I had another problem while looping over folders, looping over subfolders, and loading files from subfolders. It was terribly slow in QlikView. I ended up calling "grep" recursively over subfolders, concatenated output into a big text file and loaded that to QV. Running grep + loading one file was much faster.

You could start with "dir * > list.txt" . Probably you want to do something to content of those files later 🙂

-Alex

filelist() problem with large data amounts

saxjonas — Fri, 03 Dec 2010 14:02:41 GMT

Hi,

The files are in sub folders in the folder as well.

-----------------------

Great idea! That way I can get lots of other info as well such as owner and creation date.
I'll probably use dir then and then load the txt and later store it in a qvd.

Any tips for running dir on reload of the QVW? I know macros aren't the best things to have in the load script but maybe if they're synchronous I could do a "dir * > filelist.txt" before the LOAD statement?