Guidance on documenting ETL in QlikSense

[eugabgomes]

Hi all,

My question is not specific about Qlik development, but advice or tips on best practices.

We've been utilizing Qlik Sense for a few years in our organization. When we replaced our legacy BI Tool (which was also an ETL tool), due to the fact that Qlik has such a powerful transformation engine, we decided to utilize QlikSense as our ETL Tool as well.

We've built a layer of Extraction apps (no front-end), exporting to one QVD file per table from the DB,  a Layer of Transformation apps (also no front-end), exporting each transformed table into an individual QVD File, and several frond-end dashboards which read from these transformed QVD files with our facts and dimensions. 

This has covered our needs until now. However, now we're expanding our team of BI developers and we're facing a challenge, specially around documentation of what has been developed to share knowledge with the newcomers. 

I've been trying to generate some sort of automatized script that exports all tables and dimensions from our transformation scripts into a CSV file, so that we have some sort of initial documentation that we could populate with descriptions, etc. However, this has proven to be a challenge in terms of keeping this document up-to-date, since we are constantly adding and removing tables/dimensions/measures from our models.

This has lead us to rethink utilizing Qlik as an ETL tool and consider acquiring a standard ETL tool to transform and populate our Data Warehouse, leaving QlikSense only with the visualization part of the process.

I'd love to hear from you guys if anybody else has implemented the same strategy, using Qlik as a one-for-all tool, and if you did, how do you manage to document and share knowledge in an efficient way, while trying to minimize duplication of transformation scripts.

[Albert_Candelario · Accepted solution]

Hello @eugabgomes ,

Thanks for posting.

You would like to review the data lineage section below:

What is Data Lineage? Why You Need It & Best Practices. (qlik.com)

I think it could be a good start have a governed environment.

Cheers,

Albert

[Albert_Candelario · Accepted solution]

Also the following could be interesting:
Analyzing lineage for apps and datasets ‒ Qlik Cloud

Cheers,

Albert

[eugabgomes]

Thanks Albert, I've found that extremely useful.

I'll consider purchasing it for my organisation!

Gabriel

[Albert_Candelario]

Great Gabriel, glad you found it that useful!

Cheers,

Albert