4 Replies Latest reply: Sep 7, 2016 3:37 AM by Ricardo Jacobs RSS

    Governance Dashboard 2.01 size limitation?

    Ricardo Jacobs

      Dear all,

       

      I am trying to use the GD 2.01 to analyse the lineage of our models. The problem I have is that the Lineage-data misses some models which are present in our environment. If I narrow down the load to one specific folder I notice that all models loaded into the Lineage table are 6-30MB all models not loaded into the Lineage table are 130-850MB.

      I notice the missing files are present in the Filelist.

       

      Does any of you know if the GD limits the loading of lineage information to files smaller than a certain threshold? I tried to find it myself in the script, but I can't find any.

       

      Kind regards,

       

      Ricardo

        • Re: Governance Dashboard 2.01 size limitation?
          Tyler Waterfall

          I am not aware of any limitations to the file size of a QVW or QVD that can be "read" for lineage metadata.

          Can you try reading the metadata of one of those larger QVWs manually?

           

          To do this, create a new QV doc and in the load script click "Table File" to load. Then go to where the QVWs are and select "All Files" so you can see the QVWs. Select the QVW and in the load dialogue click on the "xml" radio button.

          This way you are loading just the XML part of the QVW.

          You should be able to save your app and reload it, seeing a bunch of metadata for that QVW.

            • Re: Governance Dashboard 2.01 size limitation?
              Ricardo Jacobs

              Hi Tyler,

               

              it appears it is coincidence it are the biggest files in that single folder. As I analyse other files which are not read there appears to be no clear reason.

              The thing I notice when loading the meta-data of the files is that they not contain any Lineage-info. I can't find a real common thing why no Lineage-info would be created. Some facts about the models:

              - The models I am looking at are dashboards( and contain data, no dropped tables)

              - The models have scheduled reload daily, when I reload them manually they produce Lineage-Info

              - The models are of any sizes

              - The models are made in different versions of Qlikview( from 8.x till most recent)

              - The models are in different folders.

              - I have admin rights in most of the models

               

              One final thought, is it possible that after a Partial Reload, no Lineage Info is created?

               

              Thanks for replying!

                • Re: Governance Dashboard 2.01 size limitation?
                  Tyler Waterfall

                  Strange indeed.

                  Can you check a couple of the qvws that are not getting metadata read and see:

                  1- Are they Partial Reload?

                  2- Can you manually inspect the metadata in the qvw by opening the .qvw with a text editor?

                   

                  Concerning the fact that the 'missing' metadata qvws showing up in the FileList - it is possible that the file ends up there because it is mentioned in reload log entry or a session log entry.

                    • Re: Governance Dashboard 2.01 size limitation?
                      Ricardo Jacobs

                      Hi Tyler,

                       

                      1. Yes, all of the files have a Daily Partial Reload and a weekly or monthly Full Reload( Partial reload is to update Section acces. Full Reload to update data)

                      2. Yes even without admin rights I can open all files with text editor or with Qlikview as xml.

                       

                      I think it shows up in the file list as other meta-data is generated when doing a Partial Reload. I think only the Lineage Information is not generated.

                       

                      In the meantime I made my own Lineage analyser which runs daily and keeps track of history. The hard thing is that we have 65+ models which all have a different schedule for full loads, so every day the situation is different. I tried to work around this by saving all Lineage in a History qvd

                      Every Day the following steps are executed:

                      1. Generate List with all files within the folders

                      2. use xml-load to recieve Lineage info for each file and Left-Join this on the file list ( now you have a list containing all files and where present Lineage Information)

                      3. Extract all File Names of which no Lineage info is found.

                      4. Retrieve Lineage information from Historical File for the list of step 3

                      5. Concatenate Lineage information from Historical with all Lineage info from step 2

                      6. Overwrite History File with table of step 5

                       

                      By doing this daily, the historical file will build up the most recent, correct Lineage information.

                       

                      Wrapping up: I think I managed to create my own Lineage Analyser. It is definitely not as elaborated as the Governance Dashboard or as the Lineage Dependency Analyser of Vasiliy Petrenko, but it does the job.

                       

                      Thanks for the suggestions and help!