Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW

Qlik Sense App: QVD Meta Data Viewer

No ratings
cancel
Showing results for 
Search instead for 
Did you mean: 
stevedark
Partner Ambassador/MVP
Partner Ambassador/MVP

Qlik Sense App: QVD Meta Data Viewer

Last Update:

Apr 23, 2018 7:50:24 AM

Updated By:

stevedark

Created date:

Apr 23, 2018 7:50:24 AM

Attachments

There is a large amount of useful information that is stored in the header of QVD files.  This gives information about the data in the file, when it was created and which app created it.  This data is in an XML format and can be viewed in a text editor.

Being XML it can also be loaded in to Sense and viewed there.

This blog post describes how this can be done and gives some background to the application uploaded here:

https://www.quickintelligence.co.uk/qlik-qvd-xml-header-viewer

The application enumerates around a number of libraries and reads the header information from each QVD in those libraries.  It persists the information it finds to a QVD and also creates a timestamped archive - so you can see how your QVDs are changing over time.

You will need to configure the app to find your libraries by editing the load script, just list the libraries in this INLINE load:

List Libraries.png

If you have any questions regarding getting this app set up please post in the comments below.

There is a QlikView version of this application also, which you can find here:

QlikView App: QVD Meta Data Viewer

Both versions of the app also include our popular Data Profiler, which you can find more information about here:

Qlik Sense App: Generic Data Profiler

I hope that you find the application useful.  You will find other applications that I have uploaded under my profile on QlikCommunity or on our Downloads page..

Steve

http://www.quickintelligence.co.uk/qlikview-blog/

Comments
pradopam
Contributor II
Contributor II

Great app!

Thanks for share 🙂 

sergeyko
Partner - Contributor III
Partner - Contributor III

Hi @stevedark ,

I am just trying this app and it gives me so much useful information on our zoo of QVDs. Thanks so much for putting it together and adapting it to Qlik Sense. You rock!

stevedark
Partner Ambassador/MVP
Partner Ambassador/MVP

Thanks @sergeyko , glad you like it.

You may also want to check my Instant Sense App, here: 
https://www.quickintelligence.co.uk/instant-qlik-sense-application/

Steve

sergeyko
Partner - Contributor III
Partner - Contributor III

This app looks awesome and I think it embodies a few key concepts that will enable us to finally start reusing master measures and dimensions across apps. This link can go a long way, thank you @stevedark !

Quick question related to this QVD metadata app - is there a view you can suggest to start monitoring changes to QVDs over time. I would be interested in tracking certain abnormalities as relates to QVDs and their fields. Some examples include QVD size changing by x%, count of unique field values changing by x%, count of blank values in a field changing by x%, etc.

I feel like I have all the data with your awesome app but I could use some help building automation to monitor all the fields across my QVDs and allow for reviewing the cases when those thresholds were exceeded.

stevedark
Partner Ambassador/MVP
Partner Ambassador/MVP

Hi @sergeyko 

You can download the app referenced in that blog post and then use that as a base for other things. I may create a metadata file for it for the QVD structure viewer at some point.

To look at changes over time you will want to add a date field to the table and then save the QVD of data with a suffix.

To add the suffix do the following:

let vSuffix = Date(today(), 'YYYYMMDD');

STORE QVDFileStructure INTO [$(vDataFolder)QVDFileStructure_$(vSuffix).qvd] (qvd);

You can then load from these with a for each vFile in FileList... or with a wildcard.

Just looking into the code for the QVF above I notice that I have put code for an incremental load into an archive QVD. You can clone this app and then build in functionality for an 'over time' version of the app. The existing measures will work wherever you have Checked Date as a dimension. For where you are not using the dimension you will always need to use the avg function, or set analysis to limit to a day at a time.

To get some ideas on how you might use set analysis to compare two points in time, to pick up % growth etc., you could take a look at this app for ideas:

https://community.qlik.com/t5/Qlik-Sense-Documents/Qlik-Sense-App-Prior-Period-Comparison-with-Set-A...

Hopefully that points you in the right direction.

Cheers,

Steve

sergeyko
Partner - Contributor III
Partner - Contributor III

@stevedark- makes sense, thanks for the pointers!

I think I have everything to build another version of the app for monitoring changes in my QVDs. I will give it a try and share the result if I am able to develop something interesting.

sergeyko
Partner - Contributor III
Partner - Contributor III

Hi @stevedark ,

I spent some more time with your app and was pleased to confirm that it is building its archive of changing QVDs and fields based on the last modification date of each QVD. These dates are embedded into QVDKey that is used for appending the data into the archive files.

Provided that most of my QVDs are update daily, it sure gives me a nice trend view over the key changes. Thanks again for sharing this immensely useful app!

The XML headers provide a host of useful information but one thing I couldn't find is the count of NULL values in fields. Have you stumbled across a solution for quickly counting  NULLs in fields across multiple data sources? I am hoping to leverage some built-in functions and/or tricks related to QVDs so I don't have to cycle through each field in each file...

0 Likes
stevedark
Partner Ambassador/MVP
Partner Ambassador/MVP

Hi @sergeyko 

Afraid I don't know of any way of doing this without loading the whole files and then interrogating the field.

Looking at @rwunderlich 's Sense Document Analyser ( https://qlikviewcookbook.com/recipes/download-info/qs-document-analyzer/ ) may be a good idea - but you will still need to load the data first.

Another option is to write a profiling step into your QVD generators, so stats are generated and written to a QVD alongside the QVD with the data in.

Hope you get to a solution that works for you.

Steve

rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP

Hi @stevedark 

I do count Nulls in my QViewer tool.  Reading the values and counting nulls is how I get the count.  

For QVFs, the engine API will report number of nulls as a TableField attribute, so you don't have to read the actual data. That's what I use in Document Analyzer. Again, this is app tables, not QVD.

-Rob

sergeyko
Partner - Contributor III
Partner - Contributor III

Thanks @stevedark & @rwunderlich !

I was hopeful that would be a shortcut for counting NULLs in QVDs without diving into each file. I will keep digging and share here if I can find anything interesting.

Version history
Last update:
‎2018-04-23 07:50 AM
Updated by: