topic Re: Allow data lineage. Please explain the value of this parameter. in Management & Governance

Allow data lineage. Please explain the value of this parameter.

korsikov — Mon, 27 Jul 2015 11:37:29 GMT

Allow data lineage this option is an analog parameter "Allow only one copy in memory"

But regardless of exposed options I have updated the data in the application. For example, changing the date of the last update in Operation monitor

Re: Allow data lineage. Please explain the value of this parameter.

Tyler_Waterfall — Thu, 30 Jul 2015 12:40:23 GMT

With 2.0.1 and 2.0.2, the "Allow data lineage" in Engine advanced settings impacts the format of any QVDs which are created in Sense server.

The difference can be seen in the attached QVDs.

Open the QVDs with a text editor and note the metadata XML section at the top.

Both QVDs include the CreatorDoc tag in the XML (line 4), but only when I had "Allow data lineage" do I have lineage information populated in the <Lineage> entity (starting in line 759 of the LineageTest_v1.qvd). This lineage information includes some of the SQL statements used to generate the table which was stored as a QVD.

My desire is to get more lineage to come in later releases of Sense server.

Re: Allow data lineage. Please explain the value of this parameter.

mountaindude — Tue, 24 Nov 2015 14:21:17 GMT

Is there a way to turn off lineage on a app-by-app basis?

I have a scenario where I create quite a few QVDs by looping over a couple of different parameters, with one QVD created in each loop iteration. Works great, but the QVDs start out at 10-15 kByte each, then keep growing to hundreds of kB each... Makes things unnecessarily slow and wastes disk space.

I will try to turn off lineage for the entire server, but I can imaging (future) scenarios where we want that enabled.

So... is it possible to turn off for specific apps, or even parts of the load script?

Re: Allow data lineage. Please explain the value of this parameter.

Tyler_Waterfall — Tue, 24 Nov 2015 14:59:16 GMT

Unfortunately, I am not aware of how this would be possible - and yes, I have experienced that with looping to load thousands of xml files.

Re: Allow data lineage. Please explain the value of this parameter.

Tyler_Waterfall — Tue, 15 Nov 2016 13:04:38 GMT

In QlikView 12 the default is Lineage off, and you have to manually set the AllowLineage in a config ini file as discussed by rwunderlich‌ here QVW LineageInfo missing

Re: Allow data lineage. Please explain the value of this parameter.

rzenere_avvale — Sun, 07 May 2017 13:07:01 GMT

Hey Tyler,

just to be sure: is this not possible yet with QS 3.2.3?

I'm facing a scenario where I need to loop many times. Allowing lineage info, the resulting QVDs are almost 20GB (all together), while without lineage info they are less than 1GB.

I would also like to understand if in QS the lineage info is used (in the operation monitor maybe) or not

Many thanks

Re: Allow data lineage. Please explain the value of this parameter.

Tyler_Waterfall — Mon, 08 May 2017 16:48:36 GMT

Hi Riccardo -

I am not surprised (though disappointed) that the QVD sizes are so much bigger when lineage is enabled in your scenario. We had a similar case where the load script loaded 140,000 xml files and with lineage enabled, the QVD included extra xml information for each of those 140,000 "source" files.

Currently there is no lineage metadata processed in the Operations Monitor Sense app.

Some Qlik Architects have been working on tools to help in situations like this, though I am not as familiar with how lineage metadata is handled. rvr‌ might have some ideas for you.

Re: Allow data lineage. Please explain the value of this parameter.

rzenere_avvale — Mon, 08 May 2017 21:27:37 GMT

Hi Tyler,

thanks for your answer.

I hope to hear from rvr‌‌ some ideas about how to handle this.

Many thanks

Re: Allow data lineage. Please explain the value of this parameter.

Renato_Vieira — Tue, 09 May 2017 16:29:16 GMT

Hello riccardozenere,

Give a quick glance at this GitHub Branch - GitHub - rvspt/QlikSenseGovernance at QlikSenseGovernance-v2

In this case, Lineage information is pulled out of Script Logs. Hope it helps!

Cheers

Re: Allow data lineage. Please explain the value of this parameter.

rzenere_avvale — Tue, 09 May 2017 17:42:19 GMT

Hey rvr‌,

really interesting project!

I'll share this one with my colleagues, I believe it will be very helpful.

Many thanks