Guidance on documenting ETL in QlikSense

eugabgomes — Thu, 03 Feb 2022 11:30:29 GMT

Hi all,

My question is not specific about Qlik development, but advice or tips on best practices.

We've been utilizing Qlik Sense for a few years in our organization. When we replaced our legacy BI Tool (which was also an ETL tool), due to the fact that Qlik has such a powerful transformation engine, we decided to utilize QlikSense as our ETL Tool as well.

We've built a layer of Extraction apps (no front-end), exporting to one QVD file per table from the DB, a Layer of Transformation apps (also no front-end), exporting each transformed table into an individual QVD File, and several frond-end dashboards which read from these transformed QVD files with our facts and dimensions.

This has covered our needs until now. However, now we're expanding our team of BI developers and we're facing a challenge, specially around documentation of what has been developed to share knowledge with the newcomers.

I've been trying to generate some sort of automatized script that exports all tables and dimensions from our transformation scripts into a CSV file, so that we have some sort of initial documentation that we could populate with descriptions, etc. However, this has proven to be a challenge in terms of keeping this document up-to-date, since we are constantly adding and removing tables/dimensions/measures from our models.

This has lead us to rethink utilizing Qlik as an ETL tool and consider acquiring a standard ETL tool to transform and populate our Data Warehouse, leaving QlikSense only with the visualization part of the process.

I'd love to hear from you guys if anybody else has implemented the same strategy, using Qlik as a one-for-all tool, and if you did, how do you manage to document and share knowledge in an efficient way, while trying to minimize duplication of transformation scripts.

Re: Guidance on documenting ETL in QlikSense

Albert_Candelario — Sat, 08 Oct 2022 16:37:13 GMT

Hello @eugabgomes ,

Thanks for posting.

You would like to review the data lineage section below:

What is Data Lineage? Why You Need It & Best Practices. (qlik.com)

I think it could be a good start have a governed environment.

Cheers,

Albert

Re: Guidance on documenting ETL in QlikSense

Albert_Candelario — Sat, 08 Oct 2022 16:39:15 GMT

Also the following could be interesting:
Analyzing lineage for apps and datasets ‒ Qlik Cloud

Cheers,

Albert

Re: Guidance on documenting ETL in QlikSense

eugabgomes — Fri, 16 Dec 2022 11:37:59 GMT

Thanks Albert, I've found that extremely useful.

I'll consider purchasing it for my organisation!

Gabriel

Re: Guidance on documenting ETL in QlikSense

Albert_Candelario — Sat, 17 Dec 2022 13:54:22 GMT

Great Gabriel, glad you found it that useful!

Cheers,

Albert

topic Re: Guidance on documenting ETL in QlikSense in Management & Governance

Guidance on documenting ETL in QlikSense

Re: Guidance on documenting ETL in QlikSense

Re: Guidance on documenting ETL in QlikSense

Re: Guidance on documenting ETL in QlikSense

Re: Guidance on documenting ETL in QlikSense