QVD files with lineageinfo data

robert_young_hc — Fri, 15 Jan 2016 15:36:09 GMT

We have a qlikview load process that is responsible for generating multiple qvd files based on a date. Each file containing only one days worth of data. The process will generate daily files from the last time it was run. What I have noticed is that when we are generating more than 1 file the size of the qvd file was growing even though the number of records was pretty static. File size started a 7k and would increase by about 3k per file.

What I found is that the QVD file is storing lineageinfo as part of the file. This lineageinfo in our case is the SQL statement used to produce the QVD. As the load progresses this lineageinfo grows as it adds new SQL statements.

We are running V11 SR13. Has anyone noticed this behavior and is there a way to turn this off.

Below is a sample of the QVD lineage info data.

First run.

- <Lineage>

- <LineageInfo>

<Discriminator>ODBC;DSN=AUTOQRY</Discriminator>

<Statement>DelayedBodies: SQL With DeptProdOFF (Fields) as (Select Fields FROM Tables WHERE DATE = '2004-04-02' GROUP BY Fields ) SELECT fields FROM Tablese GROUP BY Fields HAVING COUNT(DISTINCT field)>1</Statement>

</LineageInfo>

- <LineageInfo>

<Discriminator>ODBC;DSN=DALQRY1</Discriminator>

<Statement>Concatenate (DelayedBodies) SQL With DeptProdOFF (Fields) as (Select Fields FROM Tables WHERE DATE = '2004-04-02' GROUP BY Fields ) SELECT fields FROM Tablese GROUP BY Fields HAVING COUNT(DISTINCT field)>1</Statement>

</LineageInfo>

- <LineageInfo>

Next Run

- <LineageInfo>

<Discriminator>ODBC;DSN=AUTOQRY</Discriminator>

</LineageInfo>

- <LineageInfo>

<Discriminator>ODBC;DSN=AUTOQRY</Discriminator>

<Statement>DelayedBodies: SQL With DeptProdOFF (Fields) as (Select Fields FROM Tables WHERE DATE = '2004-04-03' GROUP BY Fields ) SELECT fields FROM Tablese GROUP BY Fields HAVING COUNT(DISTINCT field)>1</Statement>

</LineageInfo>

- <LineageInfo>

<Discriminator>ODBC;DSN=AUTOQRY</Discriminator>

<Statement>DelayedBodies: SQL With DeptProdOFF (Fields) as (Select Fields FROM Tables WHERE DATE = '2004-04-04' GROUP BY Fields ) SELECT fields FROM Tablese GROUP BY Fields HAVING COUNT(DISTINCT field)>1</Statement>

</LineageInfo>

- <LineageInfo>

<Discriminator>ODBC;DSN=DALQRY1</Discriminator>

</LineageInfo>

- <LineageInfo>

<Discriminator>ODBC;DSN=DALQRY1</Discriminator>

<Statement>Concatenate (DelayedBodies) SQL With DeptProdOFF (Fields) as (Select Fields FROM Tables WHERE DATE = '2004-04-03' GROUP BY Fields ) SELECT fields FROM Tablese GROUP BY Fields HAVING COUNT(DISTINCT field)>1</Statement>

</LineageInfo>

<Discriminator>ODBC;DSN=DALQRY1</Discriminator>

<Statement>Concatenate (DelayedBodies) SQL With DeptProdOFF (Fields) as (Select Fields FROM Tables WHERE DATE = '2004-04-04' GROUP BY Fields ) SELECT fields FROM Tablese GROUP BY Fields HAVING COUNT(DISTINCT field)>1</Statement>

</LineageInfo>

Re: QVD files with lineageinfo data

Fri, 15 Jan 2016 21:49:51 GMT

we are also splitting the qvd files daily and we didn't see size difference between qvd files.

Are you loading the qvd files incrementally or full refresh every time?

Re: QVD files with lineageinfo data

rwunderlich — Fri, 15 Jan 2016 22:03:03 GMT

This can be a problem with incremental loads. QV (correctly) accumulates the lineage. You can turn it off via easter egg (settings.ini) as shown here.

Re: Expansion of QVD file size during iterated process

-Rob

http://masterssummit.com

http://qlikviewcookbook.com

topic Re: QVD files with lineageinfo data in QlikView

QVD files with lineageinfo data

Re: QVD files with lineageinfo data

Re: QVD files with lineageinfo data