Big Data

[#1]

Hi,

My source is single table from a database having multiple columns with millions of records, data will be increase on weekly basis.

can anyone please suggest me the best data model to handle this data, is it fine to split the table and create linked table, will this approach be correct??

[jagan]

Hi,

Try Incremental loading

Incremental Load

Regards,

Jagan.

[#3]

I can implement the incremental load later but now while starting the data model how should I approach, is it fine to divide 1 source table into multiple dimension and fact tables and then using link table, can you please suggest me on data model

[Anonymous]

star scheme is preferable for the big amount of data

[#5]

so can I divide the table into dimensions and facts and then store all the keys in a linked table, is this approach good and how will be the performance...

[jagan]

Hi,

Star Schema will have better performance over single table so if possible split into dimension and fact tables.

Regards,

Jagan.

[#7]

If u hav the all flag(Insert and update) and Datemodified fields in ur fact table u can use the incremental load.

otherwise split the fact table in to several qvd using Year week, and update the last week qvd alone and concatenate the all qvd into one and use this qvd into ur document.

So that: Split into qvd like.

Fact_Table_2014-01.qvd

Fact_Table_2014-02.qvd

Fact_Table_2014-03.qvd

......

..... etc

and update the last week qvd,

Load * from Fact_Table where Yearweek(Date_field)=Yearweek(Today())

Store tab into Fact_Table_2014-03.qvd.

And Concate the all qvd into one.

Fact_Table:

Load * from Fact_Table_*.qvd

Regards,

Kabilan K.

[Anonymous]

it is not necessary use linked tables in the star scheme, but sure depends from model. linked tables is bottleneck for the big facts

[#9]

thank you so much...

[#10]

yes I will use star schema, one more suggestion required, while generating keys, which one will be better Autokey or rowno, please suggest...