topic Re: help required on the method we following for data model ?? in QlikView

help required on the method we following for data model ??

dhasharadh — Wed, 25 Nov 2020 16:16:04 GMT

we have a contract table and 3 other tables which are mixed with metrics are dimensions with an equal proportion.

data flow : from a web application - DB1 - DB2 (where ETL team prepare the tables and doing all the calculations here only).

Qlik extracts data from DB2. Now the issue is users wants the each tables divided into 2 tables like one for metrics and other for dimensions.

using the resident load we can do that and another method we are about to suggest that ETL team can divide the each table in DB2 also. which reduces the burden on Qlik.

Now which is better using the resident load or loading the tables twice or else dividing the tables in the database.

Please suggest

Re: help required on the method we following for data model ??

tresB — Tue, 01 Aug 2017 06:17:32 GMT

Dividing the tables at the source (DB) would be the best choice (in terms of performance) if you have big data set. Second best could be loading the data twice (I see this performs better than resident load while data is big). And if you have a chance (if it has majorly historical data) of creating qvd and loading from it, would be even better.

Re: help required on the method we following for data model ??

arvind_patil — Tue, 01 Aug 2017 06:27:28 GMT

HI Dhasharadh,

I suggest you can do it on database end . So you can pull it from there easily and it will save time for resident load.

Thanks,

Arvind Patil

Re: help required on the method we following for data model ??

tomasz_tru — Tue, 01 Aug 2017 06:28:26 GMT

I would do it in the database. It should be more efficient (parallel processing, smart execution plans ect.)

Tomasz

Re: help required on the method we following for data model ??

dhasharadh — Thu, 03 Aug 2017 04:40:34 GMT

Thanks Tresesco for the Quick response,

The Raw data is around >20 GB, and the ETL team is worrying about the rework that they have to do for the new design of tables.

And users want the DM schema should be like star only, is this mandatory to having a star schema.?

Re: help required on the method we following for data model ??

tresB — Thu, 03 Aug 2017 04:48:22 GMT

Star schema is not mandatory. However, if you could manage to make model as star, with such big data performance is expected to be better with star than snowflake.