Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
dhasharadh
Creator III
Creator III

help required on the method we following for data model ??

we have a contract table and 3 other tables which are mixed with metrics are dimensions with an equal proportion.

data flow : from a web application - DB1 - DB2 (where ETL team prepare the tables and doing all the calculations here only).

Qlik extracts data from DB2. Now the issue is users wants the each tables divided into 2 tables like one for metrics and other for dimensions.

using the resident load we can do that and another method we are about to suggest that ETL team  can divide the each table in DB2 also. which reduces the burden on Qlik.

Now which is better using the resident load or loading the tables twice or else dividing the tables in the database.

Please suggest

1 Solution

Accepted Solutions
tresesco
MVP
MVP

Dividing the tables at the source (DB) would be the best choice (in terms of performance) if you have big data set. Second best could be loading the data twice (I see this performs better than resident load while data is big). And if you have a chance (if it has majorly historical data) of creating qvd and loading from it, would be even better.

View solution in original post

5 Replies
tresesco
MVP
MVP

Dividing the tables at the source (DB) would be the best choice (in terms of performance) if you have big data set. Second best could be loading the data twice (I see this performs better than resident load while data is big). And if you have a chance (if it has majorly historical data) of creating qvd and loading from it, would be even better.

arvind_patil
Partner - Specialist III
Partner - Specialist III

HI Dhasharadh,

I suggest you can do it on database end . So you can pull it from there easily and it will save time for resident load.

Thanks,

Arvind Patil

tomasz_tru
Specialist
Specialist

I would do it in the database. It should be more efficient (parallel processing, smart execution plans ect.)

Tomasz

dhasharadh
Creator III
Creator III
Author

Thanks Tresesco for the Quick response,

The Raw data is around >20 GB, and the ETL team is worrying about the rework that they have to do for the new design of tables.

And users want the DM schema should be like star only, is this mandatory to having a star schema.?

tresesco
MVP
MVP

Star schema is not mandatory. However, if you could manage to make model as star, with such big data performance is expected to be better with star than snowflake.