topic Qlik data lineage in a hybrid environment in Catalog and Lineage

Qlik data lineage in a hybrid environment

sebastiannord — Thu, 30 Jun 2022 09:56:16 GMT

Hi,

We have QVD:s that are transformed in and transferred to with where they are read into the end user app.

Is it possible to view whole data flow ie QS on Windows -> Qlik Cloud? Currently we can only see them in separate views.

Example, data flow:

App A extracts data (QS on Windows).
App B transforms data on (QS on Windows).
QVD:s created by app B are transferred to Qlik Cloud using DataTransfer.
App C reads the data created by app B (Qlik Cloud).

We wish to see the complete flow from app A to C when using data lineage in app C.

Re: Qlik data lineage in a hybrid environment

Christopher_Ortega — Thu, 30 Jun 2022 12:33:34 GMT

Based on your question I assume that you are using the Qlik Lineage Connectors to see the lineage for the on-premise application.

We don't currently have the lineage from Qlik Data Transfer but it is in our plans to accomplish what you have requested.

Re: Qlik data lineage in a hybrid environment

sebastiannord — Thu, 30 Jun 2022 13:05:27 GMT

Thank you for your reply @Christopher_Ortega.

That is correct, we are using the Qlik Lineage Connectors to see the lineage for the on-premise application.

Then we just have to wait for this feature to be available in the future so we can se the complete data flow.

Re: Qlik data lineage in a hybrid environment

callanml31 — Sun, 28 Jan 2024 07:06:56 GMT

Dbt-synapse adapter works great. Not sure if you tried it. Also not sure what you mean when you say it doesn't work well with spark? The dbt-spark adapter seemed to work well for me, especially on databricks, which is unneeded now since they made their own adapter in dbt-databricks.

Re: Qlik data lineage in a hybrid environment

hermis — Sun, 28 Jan 2024 07:55:12 GMT

If you're just learning the bare minimum about dbt necessary to make it run complete replacements on your star schema, its probably worse than sql in SSIS because you won't have really learned anything yet that makes dbt useful. In my opinion you really need to be taking advantage of features like automating the merges, templating with Jina to reduce the code you type, different targets for dev and prod, tests, pre and post hooks, etc for it to be worth learning and using. Once you get ther it is, but its not superior from day 1