Empty QVWs not read by the Governance Dashboard?

[cheburashka]

Hello twa,

Am I correct that the Governance Dashboard will not scan a qvw if it does not contain data/tables in its datamodel?

Which has the consequences that my datalineage mises files read by that QVW?

,Koen

[Tyler_Waterfall]

Koen –

Not exactly.

The Governance Dashboard will scan all QVWs and return all the metadata it finds in each QVW. So, if there are any sheets, objects, fields, an owner, etc. in the QVW metadata, this will be ‘scanned’ and captured by the Governance Dashboard.

Lineage and table/field metadata, however, are only present in a QVW which has data in it.

So, if you have a QVW that loads data, stores it into a QVD or other file, and then drops all data, the lineage info will not be retained in the QVW.

The QVDs created by such QVWs would have metadata about the ‘process’ (QVW) that created them, so all is not lost. Just be sure to include the folder(s) where the resultant QVDs are created.

Tyler

[cheburashka]

Hi Tyler,

Thanks for your response. So you do not exactly analyze the script to get the meta data about the QVW tables.

That's what I was afraid for. The information about the files read by a qvw is lost for domain generators, so I cant figure out which base tables are used to create my domain tables or I would have to keep the data in the qvw, which is not a great option for storage reasons of course but I asume that will be the only possible option.

,KR Koen

[Tyler_Waterfall]

Yes, unfortunately this is the state we are in currently. Just reading the metadata from the QVW (xml part) and not trying to parse the load script. This means that just based on the QVW metadata we lose information for "QVD generator" QVWs that create a QVD(s) and then drop the table.

However, the process (QVW) that generated the QVD is captured in the QVD metadata, but not the ETL or sources used in the QVW load script to generate the QVD .

For specific database loads and web file loads, you can see parts of the load/select statement in the Governance Dashboard (again - assuming the QVW retains data).

[rwunderlich · Accepted solution]

In my experience, the Lineage metadata will not be dropped if a table created earlier is not dropped. So when full lineage is importatnt, make the first load in your script.

LineageKeeper:

LOAD 0 as LineageKeeperDummy AutoGenerate 0;

Any subsequent table, even if dropped, will show Lineage elements in the QVW. Yes, you wind up with an irrelevant table in your QVW. But if you use the same name all the time it will be obvious. Maybe we can even get QGD code to ignore them

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

[Tyler_Waterfall]

Thanks Rob for the tip!

[v_petrenko]

Another suggestion for preserving the Lineage metadata in QVD Generators is in isung a TRUNCATE approach.

That means deleting all rows from table but keeping it's structure (so it stays visible in Table Viewer). The script is simple:

Inner Join ([{Table to Truncate}])

LOAD

  *

Inline

  [

  %truncate

  ]

;

DROP Field

  [%truncate]

;

But be aware when using it inside iteration constructs like FOR...NEXT or DO...WHILE with loading-dropping! Unwanted concatenations may occur, so debug twice before making changes.

Possible side effects could be a minor increase in reload time and synthetiс keys in a data model. The latter is not important I think, since the data model is empty and no hardware resources are used. It would be just an unpleasant structure in Table Viewer.

[cheburashka]

Hi,

This is a gem.

,Thanks alot

[kevinpintokpa]

This is a major loss in functionality from Governance Dashboard 1.1 which I believe scanned all QVDs and their headers to determine lineage.

I have dozens of database table loaders that will have to be modified.  Many belong to other people, so it may not be easily feasible to modify all of them, leading of course to gaps in data.

Please consider changing the approach to scanning QVDs instead of QVWs.

[Tyler_Waterfall]

Thanks for input Kevin. I agree that capturing metadata from QVDs for lineage would be a plus. We have an open bug GOVDASH-130 which you can reference in the future. ~Tyler