topic Why is this an error in Data Model ? in App Development

Why is this an error in Data Model ?

Derek_T — Thu, 23 Jun 2022 14:26:52 GMT

Disclaimer, I'm new to Qlik and I'm trying to figure out what's wrong with such a model below.

Qlik does not like such loop, so it seems I could either use "Master Calendar" to link table 1 and 2 OR "Unique IDS".

Now let's say I just use "Master Calendar" to link the tables, if I filter Table1 on Owner A I'm still going to get all the Owner of Table 2 matching the same date.

And vice versa if I use "Unique IDS" instead I loose the connection on date.

These are standard type of query though and I'm sure there's a way to do it in Qlik but I don't know how.

Thanks !

Re: Why is this an error in Data Model ?

marcus_sommer — Thu, 23 Jun 2022 14:34:39 GMT

Do not try to link these fact-tables just concatenate (union in sql) them - maybe adding an extra source-field to be able to differentiate with selections or set analysis to from which source which data come from.

- Marcus

Re: Why is this an error in Data Model ?

MartW — Thu, 23 Jun 2022 14:39:37 GMT

Qlik uses an associative model. meaning when clicking a value other tables associated with this update with only those values.

also for good practice fieldnames need to be unique so that there are no sync tables. with regards to the looping,

try and create unique key values for your tables 1,2 and Unique IDS. after that also create those Key's for the Calender table. this should fix the problem

Re: Why is this an error in Data Model ?

Derek_T — Thu, 23 Jun 2022 14:39:23 GMT

Thanks Marcus, I think that's the part I don't understand would you have any link to refer me to please ?

Coming from PowerBI I thought this was how the data model worked, but I have the feeling it's not the exact same logic here.

Re: Why is this an error in Data Model ?

Derek_T — Thu, 23 Jun 2022 14:45:16 GMT

Yes that's what I did, sorry it does not show in my mock up example. But in Qlik Table1 and Table2 have column Date1 and Date2 (same for ID) and the Master calendar as column Date1 and Date2 as well.
So they are unique keys and Qlik is not trying to associate Table1 and Table2 (with synthetic key on Date and ID.)

But even then, it still does not like the that Table1 and Table2 are connected through 2 different tables. Whereas it makes perfect sense to me to be able to filter on a date range AND an ID (or Owner).

It's been a couple of years since I used PowerBI but to my recollection these kinds of connections were allowed in the model.

Re: Why is this an error in Data Model ?

MartW — Thu, 23 Jun 2022 14:47:25 GMT

@Derek_T, can you share a screenshot of the datamodel viewer?

Re: Why is this an error in Data Model ?

marcus_sommer — Thu, 23 Jun 2022 15:05:54 GMT

Like you have already observed a key on ID or Date wouldn't associate all table-parts else just the ones on which they are linked. A possible bypass would be to combine both fields within a single field for the association and renaming the origin fields to avoid the creation of synthetic keys or circular references.

But it's often not enough because now you couldn't access both fields within the UI easily because they have different names and also connecting these tables with other dimension-tables.

Further linking tables didn't solve any challenges if the relation-ship between the tables isn't 1:1 and/or if there are missing keys on one or even on both sides. Quite often it ends in creating a linking table between the fact-tables which contains the distinct concatenated keys from all fact-tables and on which the further dimensions are linked.

By larger datasets this could become a performance-problem because such link-tables could grow much bigger as the fact-tables itself. Therefore why doing it more complicated as just concatenating the facts directly? Within the most scenarios it worked very well, is simple to create and leads to the officially recommended star-scheme data-model (as best compromise between efforts, maintainability and performance).

- Marcus

Re: Why is this an error in Data Model ?

Derek_T — Thu, 23 Jun 2022 15:17:56 GMT

Here it is :

Re: Why is this an error in Data Model ?

Derek_T — Thu, 23 Jun 2022 15:48:38 GMT

Thank you, does that mean Qlik data Model does not support 1:n or n:n relationship ?

Re: Why is this an error in Data Model ?

marcus_sommer — Thu, 23 Jun 2022 16:10:56 GMT

Of course it's possible to build valide data-models with such relationships but depending on the data and the requirements it might not always be trivial to merge them with join-approaches in the script and/or associating those tables in the data-model. Especially if there are also the mentioned missing keys - which would mean the attempt to link a value against NULL which couldn't work. That's not seldom, for example if you have sales without budgets or reverse - you won't be able to get a complete picture of them if any keys points against NULL.

By simply adding such datasets with a concatenating respectively union all the linking-challenges wouldn't be there - neither between the fact-tables nor in regard to associate any dimensions. It's really so very simple.

- Marcus

Re: Why is this an error in Data Model ?

Derek_T — Wed, 29 Jun 2022 14:12:41 GMT

I found a solution to my problem:

I created a "Link" table using Unique IDs and Master Calendar on which all tables are connected.

Load Distinct
ID
resident Unique IDs;

Join
Load Distinct
DATE
Resident Master_Calendar;

And created the Key = ID & '-' DATE in the Link Table as well as Table1 and Table2.

Only issue I have with such an approach is that the Link Table size grow very quickly, especially if such a key involve more than 2 fields. So I'm still open to suggestion if there's a more efficient approach, I don't know if the Link table is stored in memory are recomputed every time it's queried.

Re: Why is this an error in Data Model ?

marcus_sommer — Wed, 29 Jun 2022 15:12:46 GMT

The link-table is stored within the file and is also kept in memory and is used in all related UI objects to create the appropriate dimensional context within virtual tables on which the final aggregations are performed.

In general such approach worked but from a performance point of view it's by larger data-sets usually not the best choice. Therefore my above suggestion not to create a link-table else to concatenate the fact-tables directly.

- Marcus

Re: Why is this an error in Data Model ?

Derek_T — Wed, 29 Jun 2022 18:37:09 GMT

Thanks @marcus_sommer , would you have some code examples so I understand what you mean by concatenating the tables please ?

In my simple example above Table 1 and 2 have the same 3 fields but in the real use case each table have around 30 fields each and most of them are completely different. So would concatenation still be a viable solution in this case ?

Re: Why is this an error in Data Model ?

marcus_sommer — Thu, 30 Jun 2022 08:13:49 GMT

Meant was something like:

t: load F1, F2, F3, F4, F5, F6, 'X' as F10 from X;
concatenate(t) load F1, F2, F3, F7, F8, F9, 'Y' as F10 from Y;

That the table is then more or less asynchron is neither a technically problem nor must it cause performance or logically issues. How suitable it is in regard to the degree of asynchronous and/or to the size of the data-set and/or the UI requirements depends - especially compared to any alternatives and the efforts and side-effects which come with them.

The above suggestion is further only the starting point to develop a data-model and doesn't mean that no further transformations are needed or at least sensible. For example it's possible and recommended to synchronize as many as possible of the field-names and their content, like SalesDate and BudgetDate which aren't not really different information and could be just renamed as Date - the needed context could you get from the source-information. Also measures like Sales and Budget could be kept within a single field - by many measure-fields you may further consider to transform the cross-table logic of them into a data-stream which means to create two fields for them - KPI & Value.

This leads to rather long and small tables which may potentially have slight RAM and UI performance disadvantages against short and wide tables - but for us it worked very well and we use this approach nearly everywhere and we could improve our handling + performance significantly.

Beside this some of the missing/wrong field-values might be cleaned/filled/prepared and/or outsourced in further dimension-tables and/or ...

- Marcus