Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
korsikov
Partner - Specialist III
Partner - Specialist III

Allow data lineage. Please explain the value of this parameter.

Allow data lineage this option is an analog parameter "Allow only one copy in memory"

But regardless of exposed options I have updated the data in the application. For example, changing the date of the last update in Operation monitor

1 Solution

Accepted Solutions
Tyler_Waterfall
Employee
Employee

With 2.0.1 and 2.0.2, the "Allow data lineage" in Engine advanced settings impacts the format of any QVDs which are created in Sense server.

The difference can be seen in the attached QVDs.

Open the QVDs with a text editor and note the metadata XML section at the top.

Both QVDs include the CreatorDoc tag in the XML (line 4), but only when I had "Allow data lineage" do I have lineage information populated in the <Lineage> entity (starting in line 759 of the LineageTest_v1.qvd). This lineage information includes some of the SQL statements used to generate the table which was stored as a QVD.

My desire is to get more lineage to come in later releases of Sense server.

View solution in original post

9 Replies
Tyler_Waterfall
Employee
Employee

With 2.0.1 and 2.0.2, the "Allow data lineage" in Engine advanced settings impacts the format of any QVDs which are created in Sense server.

The difference can be seen in the attached QVDs.

Open the QVDs with a text editor and note the metadata XML section at the top.

Both QVDs include the CreatorDoc tag in the XML (line 4), but only when I had "Allow data lineage" do I have lineage information populated in the <Lineage> entity (starting in line 759 of the LineageTest_v1.qvd). This lineage information includes some of the SQL statements used to generate the table which was stored as a QVD.

My desire is to get more lineage to come in later releases of Sense server.

mountaindude
Partner Ambassador
Partner Ambassador

Is there a way to turn off lineage on a app-by-app basis?

I have a scenario where I create quite a few QVDs by looping over a couple of different parameters, with one QVD created in each loop iteration. Works great, but the QVDs start out at 10-15 kByte each, then keep growing to hundreds of kB each... Makes things unnecessarily slow and wastes disk space.

I will try to turn off lineage for the entire server, but I can imaging (future) scenarios where we want that enabled.

So... is it possible to turn off for specific apps, or even parts of the load script?

Please mark the post as a solution if it provided you with a solution to the topic at hand. Thanks!
Tyler_Waterfall
Employee
Employee

Unfortunately, I am not aware of how this would be possible - and yes, I have experienced that with looping to load thousands of xml files.

Tyler_Waterfall
Employee
Employee

In QlikView 12 the default is Lineage off, and you have to manually set the AllowLineage in a config ini file as discussed by rwunderlich‌ here QVW LineageInfo missing

rzenere_avvale
Partner - Specialist II
Partner - Specialist II

Hey Tyler,

just to be sure: is this not possible yet with QS 3.2.3?

I'm facing a scenario where I need to loop many times. Allowing lineage info, the resulting QVDs are almost 20GB (all together), while without lineage info they are less than 1GB.

I would also like to understand if in QS the lineage info is used (in the operation monitor maybe) or not

Many thanks

Tyler_Waterfall
Employee
Employee

Hi Riccardo -

I am not surprised (though disappointed) that the QVD sizes are so much bigger when lineage is enabled in your scenario. We had a similar case where the load script loaded 140,000 xml files and with lineage enabled, the QVD included extra xml information for each of those 140,000 "source" files.

Currently there is no lineage metadata processed in the Operations Monitor Sense app.

Some Qlik Architects have been working on tools to help in situations like this, though I am not as familiar with how lineage metadata is handled. rvr‌ might have some ideas for you.

rzenere_avvale
Partner - Specialist II
Partner - Specialist II

Hi Tyler,

thanks for your answer.

I hope to hear from rvr‌‌ some ideas about how to handle this.

Many thanks

Renato_Vieira
Employee
Employee

Hello riccardozenere,

Give a quick glance at this GitHub Branch - GitHub - rvspt/QlikSenseGovernance at QlikSenseGovernance-v2

In this case, Lineage information is pulled out of Script Logs. Hope it helps!

Cheers

rzenere_avvale
Partner - Specialist II
Partner - Specialist II

Hey rvr‌,

really interesting project!

I'll share this one with my colleagues, I believe it will be very helpful.

Many thanks