topic Re: Big Data in QlikView

Big Data

Fri, 19 Sep 2014 07:21:06 GMT

Hi,

My source is single table from a database having multiple columns with millions of records, data will be increase on weekly basis.

can anyone please suggest me the best data model to handle this data, is it fine to split the table and create linked table, will this approach be correct??

Re: Big Data

jagan — Fri, 19 Sep 2014 07:28:25 GMT

Hi,

Try Incremental loading

Incremental Load

Regards,

Jagan.

Re: Big Data

Fri, 19 Sep 2014 07:33:39 GMT

I can implement the incremental load later but now while starting the data model how should I approach, is it fine to divide 1 source table into multiple dimension and fact tables and then using link table, can you please suggest me on data model

Re: Big Data

Anonymous — Fri, 19 Sep 2014 07:36:24 GMT

star scheme is preferable for the big amount of data

Re: Big Data

Fri, 19 Sep 2014 07:45:13 GMT

so can I divide the table into dimensions and facts and then store all the keys in a linked table, is this approach good and how will be the performance...

Re: Big Data

jagan — Fri, 19 Sep 2014 07:46:08 GMT

Hi,

Star Schema will have better performance over single table so if possible split into dimension and fact tables.

Regards,

Jagan.

Re: Big Data

Fri, 19 Sep 2014 07:47:10 GMT

If u hav the all flag(Insert and update) and Datemodified fields in ur fact table u can use the incremental load.

otherwise split the fact table in to several qvd using Year week, and update the last week qvd alone and concatenate the all qvd into one and use this qvd into ur document.

So that: Split into qvd like.

Fact_Table_2014-01.qvd

Fact_Table_2014-02.qvd

Fact_Table_2014-03.qvd

......

..... etc

and update the last week qvd,

Load * from Fact_Table where Yearweek(Date_field)=Yearweek(Today())

Store tab into Fact_Table_2014-03.qvd.

And Concate the all qvd into one.

Fact_Table:

Load * from Fact_Table_*.qvd

Regards,

Kabilan K.

Re: Big Data

Anonymous — Fri, 19 Sep 2014 07:48:27 GMT

it is not necessary use linked tables in the star scheme, but sure depends from model. linked tables is bottleneck for the big facts

Re: Big Data

Fri, 19 Sep 2014 08:31:47 GMT

thank you so much...

Re: Big Data

Fri, 19 Sep 2014 08:33:46 GMT

yes I will use star schema, one more suggestion required, while generating keys, which one will be better Autokey or rowno, please suggest...

Re: Big Data

jagan — Fri, 19 Sep 2014 08:43:14 GMT

Hi,

If you don't have key fields then don't split the table maintain it in a single table.

For example:

If you have

CountryID, Country, Date, Sales then split Country dimension table, if your data not in this format then do it with a single table. Don't complicate the model.

Regards,

Jagan.

Re: Big Data

Fri, 19 Sep 2014 09:12:41 GMT

yes I am splitting the one with key fields remaining I am maintaining in same table...suggest me which one is suited for this scenario autokey or rownumber??

Re: Big Data

jagan — Fri, 19 Sep 2014 09:28:51 GMT

No need o Autokey just use existing primary key fields of Dimension tables.

Regards,

Jagan.