Metadata Management and Data Dictionaries

[#1]

I'm wondering how business users / BI Analysts manage their metadata. I don't actually mean ETL diagrams, table schemas, or traditional database metadata etc. I mean business definitions, formulas, and process work-flows that is accessible and meaningful to the end-user.

A similar question was asked by Danielle Brown in this thread: Is it possible to create a data dictionary with the Data Governance app? almost a year ago, and the response was to use QlikView Expressor (which is really geared towards IT) or use Excel / MS Access databases. The continued problem is the need to maintain and update the database whenever something changes. In my experience, documentation is generally the last thing to be updated, and is often relegated to the end of the project life-cycle, at which time most people are gearing up for the next project and are disinterested in maintaining a business glossary of definitions, formulas, and most importantly, workflow changes.

Is this form of (business) Metadata Management a problem for most companies? How have others approached solving this problem?

Thanks,

- A.

[Michael_Tarallo]

Hello Abhijeet,

I assume you mean in regards to Sense and non QlikView, correct?

If Sense, then the Master Library would the current approach, defined within a Sense app.

In regards to "business users" (non-IT) managing metadata - the current approach, (which has plans to be improved over time) - is to use the Master Library. The Master Library is specific to a Sense app only, and allows one to define dimensions, measures and even visualizations that are to be used within the app. It can also be used to create a template or shell (no visualizations, just a data model and data) with the most commonly used definitions for dimensions and measures.

Qlik products traditional metadata (or semantic layer) approach for a more technical crowd is to leverage QlikView Expressor or the QlikView script syntax to create a QVD file. The QVD file is a reusable representation of the in-memory data model stored on disk.  This QVD can store the expressions and measures to be used and shared across the deployment. Maintaining it from the IT perspective is either done through the visual representation tool in QlikView Expressor or through script using the syntax.

I would suggest check out some of these videos here: New to Qlik Sense Videos  and look at  Creating a Data Model in Qlik Sense (video) and  Using The Master Library (video) to get an idea pf the process.

If you would like to discuss this further, please let us know and we can put you in touch with the appropriate people.

If you can, please mark this discussion appropriately, if I have answered your question to your satisfaction.

Kinds regards,

Mike

Qlik

[mr_barriesmith]

In QlikView, the .QVW was open and we could read the file as XML.  The metadata exposed included everything from Tables and Fields to Sheet objects and Expressions.

I would really appreciate this open metadata in Qlik Sense.  The master list is neat but I can't get to it and so I can't do cross app analysis in terms of consistent business labels and expressions.

[Michael_Tarallo]

Hello Adam, that type of work is coming to Qlik Sense. Currently we are enabled to monitor and report on system related usage / resources within our Governance section of the Sense platform - soon to be released this month. Those capabilities will be expanded upon to include metadata from apps such as the expressions, measures, dimensions etc. and much more.

Stay tuned!

[#5]

Hi Michael,

Thanks for the response. The Master Library feature is pretty neat, and sort of gets at what I'm looking for.

Is the master-library tied to a data model (star schema) or is it tied to a specific app (dashboard)? So, say I make one star schema that can be loaded into different dashboards, does this "Master Library" become available in all those dashboards? Or does it need to be defined for each one?

Also, is the Master Library maintained by the end-users, or is it maintained by IT / the BI team?

Is it possible to bulk load information into the library from, say, an excel file or database, rather than manually re-typing the expressions and definitions into QlikSense? Is QlikView Expressor for QV11 similar in functionality to the Master Library?

Finally, is there any way to track that the description/expression changed, who changed it and when/why?

Thank you for your help!

Ab.

[Michael_Tarallo]

Hello Abhijeet - I just wanted to let you know that I got the reply and that I will take a look at these question and I will respond shortly.

Mike

[Michael_Tarallo · Accepted solution]

Hey Abhijeet - here are my replies, some may require further investigation and spark new discussion. Please post new questions in the forum, so others can have the opportunity to assist,

(Q) Is the master-library tied to a data model (star schema) or is it tied to a specific app (dashboard)?

(A) Technically, In the current version, the Master Library is tied to an App in Qlik Sense, which is tied to the data model that is loaded into the App.

Here are the most common steps when creating a Qlik Sense app:

• Create a New App
• Access data using a variety of connectors with the Data Load Editor or files using Quick Data Load
• If loading more than one source of data, columns with the same name will be joined together by default using a Full Outer Join
• If any transformation is needed on the data - the Qlik syntax can be used
• A "data model" is created as part of the APP
• In the Data Model Viewer you can assign certain columns to be dimensions or measures using aggregates
• These Dimensions and Measures will automatically display in the design mode in the app sheet in the Master Library
• In the Master Library and Field list - you can further define and add more measures and dimensions
• The library is only available in that APP
• This APP can contain just the data model and be used as a template for users to create Ad hoc content from.

(Q) So, say I make one star schema that can be loaded into different dashboards, does this "Master Library" become available in all those dashboards? Or does it need to be defined for each one?

(A) In an APP - you can have many (dashboards) sheets that use the same data model. You can then further define those apps with security and make them accessible in App groups known as Streams, to further control user access. Think of he APP as your metadata models (in a traditional BI sense).

(Q) Also, is the Master Library maintained by the end-users, or is it maintained by IT / the BI team?

(A) They can be maintained by individuals (not necessarily IT) that you designate as "Content Admins". It will depend on the complexity of the data being loaded and how much custom work via expressions and transformation may be needed.

(Q) Is it possible to bulk load information into the library from, say, an excel file or database, rather than manually re-typing the expressions and definitions into Qlik Sense?

(A) Simply stated with our APIs the answer is yes. We have a host of APIs that can automate many manual tasks. The Qlik Engine API has a "Generic Object Definition" - where you can define a "hypercube" as they call it using a number of methods. You would define these in a standard JSON structure. There is an example in the Sense platform documentation that customers will be able to view once the product is GA later this month.

(Q) Is QlikView Expressor for QV11 similar in functionality to the Master Library?

(A) - No - simply stated QlikView Expressor allows you to prepare a QVD file for QlikView to load. A QVD is a uncompressed representation of the in-memory data model on disk. QlikView Expressor provides a visual approach to accessing and transforming data along with tools to manage and reuse the process. - These are two completely different products. QVDs created by QlikView Expressor CAN be used in Qlik Sense - however once loaded into Sense they will be converted to the newer columnar format that Sense uses.

(Q) Finally, is there any way to track that the description/expression changed, who changed it and when/why?

(A) This is something we are looking into with our new governance and data management tools/approaches. Currently not in the 1.0 version of Sense.

Hope this is helpful

Please mark the appropriate replies as helpful / correct so our team and other members know that your question(s) has been answered to your satisfaction.

Regards,

Mike Tarallo

Qlik

@mtarallo