Qlik Community

Qlik Sense Deployment & Management

Discussion board where members learn more about Qlik Sense Installation, Deployment and Management.

korsikov
Valued Contributor II

Allow data lineage. Please explain the value of this parameter.

Allow data lineage this option is an analog parameter "Allow only one copy in memory"

But regardless of exposed options I have updated the data in the application. For example, changing the date of the last update in Operation monitor

1 Solution

Accepted Solutions
Employee
Employee

Re: Allow data lineage. Please explain the value of this parameter.

With 2.0.1 and 2.0.2, the "Allow data lineage" in Engine advanced settings impacts the format of any QVDs which are created in Sense server.

The difference can be seen in the attached QVDs.

Open the QVDs with a text editor and note the metadata XML section at the top.

Both QVDs include the CreatorDoc tag in the XML (line 4), but only when I had "Allow data lineage" do I have lineage information populated in the <Lineage> entity (starting in line 759 of the LineageTest_v1.qvd). This lineage information includes some of the SQL statements used to generate the table which was stored as a QVD.

My desire is to get more lineage to come in later releases of Sense server.

9 Replies
Employee
Employee

Re: Allow data lineage. Please explain the value of this parameter.

With 2.0.1 and 2.0.2, the "Allow data lineage" in Engine advanced settings impacts the format of any QVDs which are created in Sense server.

The difference can be seen in the attached QVDs.

Open the QVDs with a text editor and note the metadata XML section at the top.

Both QVDs include the CreatorDoc tag in the XML (line 4), but only when I had "Allow data lineage" do I have lineage information populated in the <Lineage> entity (starting in line 759 of the LineageTest_v1.qvd). This lineage information includes some of the SQL statements used to generate the table which was stored as a QVD.

My desire is to get more lineage to come in later releases of Sense server.

Luminary
Luminary

Re: Allow data lineage. Please explain the value of this parameter.

Is there a way to turn off lineage on a app-by-app basis?

I have a scenario where I create quite a few QVDs by looping over a couple of different parameters, with one QVD created in each loop iteration. Works great, but the QVDs start out at 10-15 kByte each, then keep growing to hundreds of kB each... Makes things unnecessarily slow and wastes disk space.

I will try to turn off lineage for the entire server, but I can imaging (future) scenarios where we want that enabled.

So... is it possible to turn off for specific apps, or even parts of the load script?

Employee
Employee

Re: Allow data lineage. Please explain the value of this parameter.

Unfortunately, I am not aware of how this would be possible - and yes, I have experienced that with looping to load thousands of xml files.

Employee
Employee

Re: Allow data lineage. Please explain the value of this parameter.

In QlikView 12 the default is Lineage off, and you have to manually set the AllowLineage in a config ini file as discussed by rwunderlich‌ here QVW LineageInfo missing

riccardozenere
Contributor III

Re: Allow data lineage. Please explain the value of this parameter.

Hey Tyler,

just to be sure: is this not possible yet with QS 3.2.3?

I'm facing a scenario where I need to loop many times. Allowing lineage info, the resulting QVDs are almost 20GB (all together), while without lineage info they are less than 1GB.

I would also like to understand if in QS the lineage info is used (in the operation monitor maybe) or not

Many thanks

Employee
Employee

Re: Allow data lineage. Please explain the value of this parameter.

Hi Riccardo -

I am not surprised (though disappointed) that the QVD sizes are so much bigger when lineage is enabled in your scenario. We had a similar case where the load script loaded 140,000 xml files and with lineage enabled, the QVD included extra xml information for each of those 140,000 "source" files.

Currently there is no lineage metadata processed in the Operations Monitor Sense app.

Some Qlik Architects have been working on tools to help in situations like this, though I am not as familiar with how lineage metadata is handled. rvr‌ might have some ideas for you.

riccardozenere
Contributor III

Re: Allow data lineage. Please explain the value of this parameter.

Hi Tyler,

thanks for your answer.

I hope to hear from rvr‌‌ some ideas about how to handle this.

Many thanks

Employee
Employee

Re: Allow data lineage. Please explain the value of this parameter.

Hello riccardozenere,

Give a quick glance at this GitHub Branch - GitHub - rvspt/QlikSenseGovernance at QlikSenseGovernance-v2

In this case, Lineage information is pulled out of Script Logs. Hope it helps!

Cheers

riccardozenere
Contributor III

Re: Allow data lineage. Please explain the value of this parameter.

Hey rvr‌,

really interesting project!

I'll share this one with my colleagues, I believe it will be very helpful.

Many thanks

Community Browser