Newbie to QlikView

[#1]

Hi All,

I am Karthik, 11+ years of experience in IT industry. I am working as a Database Designer/ETL Developer(Datastage) in India. I have started to learn qlikview (installed in my home laptop) from yesterday and saw some videos (youtube), visited some tutorial, created some basic reports with the help of video. I would say I am newbie to QlikView, So Incase if my questions are immature, please answer without getting bored

I have some requirement as below

Step 1: Need to load two files from DB to Qliview QVD file.

            a) File 1 - Truncate and Load - 150 GB data - Daily Morning

            b) File 2 - Incremental Load - 100 GB Data - Daily Evening

Step 2: Incremental Load window is scheduled on every day night

Step 3: Have written some validations, functions, aggregate functions just by joining File 1 (QVD) and File 2(QVD).

My Questions are

1)How much RAM/Disk memory is required for doing step 1 ?

2)What kind of steps needs to be taken care for loading both the file? I am looking from a Admin perspective.

3)is Index concept available in QlikView ? Because While joining QVD1 & QVD2, can I create some indexes for the joining columns?

4)is 'Table Scan' ( DB Term) applied while accessing QVD file in any scenario? or is there any index based search available? If yes, is B-Tree methodology is used or something different?

5)If 'Aggregate' perform slow, what is the alternate way to achieve? may be moving it to DB or ETL is an option, but if I want to fine tune it in BI itself, how to achieve?

6)is QlikView Optimizer designed based on 'SET' theory or 'Row by Row' ?

7)How to see the query plan in QlikView?

😎 Qlikview Server is required to create/publish the dashboard? Right? If yes, I don't see it in 'Free' download. How much cost to buy QlikView Server for my home laptop ( Personal Edition) ?

Expert Inputs are welcome!

[marcus_sommer]

Do you mean really GB of data - 150 GB = 150.000.000.000 Byte? If yes you will need a lot of RAM and also the other hardware-parts in your environment must be quite large.

Not really clear is what do you mean with incremental loading 100 GB and that both files should be joined - is really a join-statement meant or some kind of updating or an union (concatenate in qlik term)?

From the size of raw-data couldn't be derived the size in qlik - it is depending on the kind of data especially if they are numeric or strings and even more important how many distinct values has each field.

I think you should give more details to your data before going further is sensible.

- Marcus

[#3]

150 GB - Just for example...As I said, I narrated the above just for scenario to understand the RAM requirement.

I can see my DB size as 45 GB... How much RAM is required?

100 GB - Again, The DB data size is 100 GB, As part of incremental load we may expect 500 MB data incremental data on daily basis.

Assume, Both of these are stored in a seperate QVD...If I want to concatenate both the QVD file, how much RAM cost

is required?

Any answer for my questions?

[marcus_sommer]

You couldn't seriously derive the needed amount of RAM from the size of the database or the size from a flat-file. Qlik used for each field a symbol-table which only stored the distinct values from this field and those fields are linked to a data-table which only contained binary pointers to the values in the system-tables.

This meant the kind of data is essential to how many RAM is needed within qlik - whereby most important is the number of distinct field-values within each field. With "common/normal" business-data the needed size reduced often to 20% and lesser but it always depends. Take a look into the following postings to get a better understanding on how qlik handled the data-processing:

Symbol Tables and Bit-Stuffed Pointers

Symbol Tables and Bit-Stuffed Pointers - a deeper look behind the scenes

ByteSizes from Values in QlikView‌

Another very important point is to load only those fields which you are really need for the requirred calculations and views. Most databases contain a lot of fields which are only needed within them and it made not much sense to transfer them into qlik, for example fields with record-id's and similar stuff (only if there are any issues with the number of records with or without any filtering or with other data-quality issues you might load them and/or trace them with recno() or rowno() an the qlik-side and compare the records - but this are just temporary development steps).

The easiest way to get a "feeling" for it would be to just load (all or maybe a rather typical part of) the data and to store them as qvd. Further it would be quite helpful if you provides more details like the number of records and fields and which type of data are included than otherwise it's a quite generic discussion.

- Marcus

[#5]

Thanks Marcus for your response. I'm sorry..as i was bit busy with some data modeling stuff for the last 2 weeks, I didn't spend time to learn qlikview further more...I have some breathing period now..Let me read the attached post and come back....

Can I get the qlikview server - personal edition ? how much cost in INR ?

I am working as a data modeler in my organization ( Oracle/SQLServer/Sybase - DB Technologies) , Datastage & some knowledge on Talend ( ETL Technologies)... I found a QlikView Architect position in my company yesterday.

I am interested to move myself to this role..although as i am beginner to qlikview, i am bit afraid as it is "Architect" role...

what would be the generic expectation from a qlikview architect? Once I learn Qlikview in another 1 month, can i move to this kind of role ?  or qlikview developer would be fine to start in BI area? 

[marcus_sommer]

The server isn't available as personal edition so your company will need to purchase one.

I'm not sure if anyone understands the same by the term QlikView Architect. I would tend to interpret this as someone who create the concepts, the data-architecture, the data-models and runs them for a whole qlikview environment with maybe hundreds/thousands (by larger ones) of users and applications in a clusterred environment.

That's far away from trivial and I think a few months of working won't be enough experience to fullfill these role even if you have a good deal of experience with databases and ETL technologies. Of course they are very useful for this role but qlikview worked different to them in some way and it will be rather painful to try to implement the well-working logic from the databases directly to qlikview.

Here is (beside this community) a good way to learn how qlik worked: Books and literature.

- Marcus