Qlikview Desktop Edition for Load Testing

[#1]

Hi, I am receiving around 70GB data every 15 minutes in the form of several text files. And I wish to use qlikview to correlate them and combine them into one single text file. Given such large amount of data, can anyone tell me if qlikview is able to accomplish everthing within 15 minutes? If so, how should the server specs be? For testing purpose, does it make any difference if i use a qlikview desktop edition instead of purchasing a server edition?

[#2]

Anyone has any experience dealing such large amount of data with Qlikview?

[Miguel_Angel_Baeyens]

Hi,

Definitely, a laptop with the Desktop is not the best way to test a real life scenario. How much RAM does it uses opening one single text file? You certainly will compress and save some memory and disk usage storing those files into QVD files, but before doing that you need to put into memory the whole file.

It's not all about the hardware. Of course, the better and faster RAM you have and the faster Intel CPU you have the better performace you will get. But it will depend on how many files do you need to load, how many analyses and charts do you need to draw... I mean, does it make sense to have ALL rows and serve them to the users? My guess is that it is not.

Can you elaborate on your actual requirements?

Miguel

[#4]

Hi Miguel,

     Thank you for the reply. All the source files are stored in a remote storage. I assume it will be big enough to accomodate all the files. So let us not consider disk space as an issue. My concern is memory and processing speed. There are 4 source text files all together. One static file with 2 million records. The other three with more than that(They are dynamically generated). I am not considering drawing charts now. All i need to do is load those 4 text files into memory, join them, and export the joined table into a text file. As I have mentioned, the size of all 4 files could amount to as large as 70GB. And I need to finish processing them within 15 minutes because another batch of files will be arriving the next 15 minutes.

Regards,

Xue Bin

[Miguel_Angel_Baeyens]

Hi Xue Bin,

Then I don't recommend you QlikView at all to perform those tasks. You can append files using Windows batch commands, not to mention any other software more suitable for appending or concatenating files.

Hope that helps.

Miguel

[#6]

Hi Miguel,

     Sorry I still didn't make myself clear enough. What I need to deal with those 4 files is to associate them. I give a brief example here.

File1.csv

CustId, CustNum

1,1001

2,1002

...

File2.csv

CustNum, Location, Time

1001,New York,08-09-2012

1002,LA,09-08-2012

1001,Stuttgart,11-11-2012

...

Combined file should look like this:

CustId,CustNum,Location,Time

1,1001,New York,08-09-2012

1,1001,Stuttagrt,11-11-2012

2,1002,LA,09-08-2012

...

That's basically what I am trying to achieve here.

Best Regards,

Xue Bin

[Miguel_Angel_Baeyens]

Hi,

Got it now, thanks. Still, I'd recommend you, if there are not going to be any analyses or charts, importing those files into a MySQL or SQL Server Express, so you can store the info and update when needed and dump or export them once linked to a new file. I don't see here the advantage of using QlikView for the sole reason that is fast... QlikView is indeed fast, but it all depends on your hardware and what do you want to get.

In any case, the hardware requirements to fit your needs are quite demanding, regardless the tool you want to use.

Hope that helps.

Miguel

[#8]

Thanks Miguel,

     The reason I am not considering drawing charts now is because I am not quite sure if Qlikview is able to accomodate such large amount of data. Given 70GB every 15 minutes, the data could amount to 50TB in one week. I am pretty sure it's impossible to load one week's data into Qlikview. That's why I'm trying to process data every 15 minutes and split the data into smaller qvd file. After that's done, I plan to do some trend analysis. Again, size of the data could become an issue. I have to carefully select fields so that the total data loaded would not get too big to handle. In fact, I still prefer to use Qlikview to present the data as it's so flexible, intuitive and easy to use. It's such a nice tool:)

Regards,

Xue Bin