The lineage view in Qlik Cloud Analysis is an excellent tool. However, after adjusting our approach to follow an example provided to us by Qlik, I am finding that the Lineage view is not showing the complete picture any more. Are there limitations or patterns that would affect how much data can be displayed in the lineage view?
Environment before adjustments:
- Separate spaces for Extract Dev, Extract UAT, Extract Prod, Transform Dev, Transform UAT, Transform Prod.
- QVDs were stored in a single S3 bucket, using a shared data connection stored in a space and referenced by all load scripts, regardless of which space it was being executed from.
- A config.xlsx file was used to define the space as DEV, UAT, or PROD and pointed to the correct folder inside the S3 bucket.
- This setup provided a complete view when an analysis app was built, so that we could see all the way back to the Extract app, and listed the individual QVDs in each.
- However, it required maintaining the 'config' file, which I was not happy with because once you upload it, you can't download it. And the example provided by Qlik had separate connections in each space which would leverage the relative paths without depending on the config file.
- The connections in the script were referenced directly (as in From [lib://TransformProd:DataFiles/data.qvd]) and only the folders were defined by variables.
- In the apps at all levels, we saw the data lineage back to the original source, with the QVDs listed.
Environment after the changes were made:
- The same spaces exist for Extract Dev, Extract UAT, Extract Prod, Transform Dev, Transform UAT, Transform Prod.
- QVDs are now stored in 3 separate S3 buckets, one each for DEV, UAT, and PROD. The data connections are now repeated in each space and point to the correct bucket as needed.
- Because the connections are now fully relative, we don't need a config file.
- The connections in the script are referenced by variables.
- After doing reloads on the apps all the way up to the analysis app, we still don't see any of the QVDs listed in the lineage, and don't see the original source. Nor do I see back to the original apps.
Is this because we are no longer using a single shared connection across the spaces? I have not been able to find anything online that would speak to this limitation. And if so, why would Qlik suggest a pattern that would invalidate the Lineage view?
