topic Re: QDC Dataflow Dependencies in Catalog and Lineage

QDC Dataflow Dependencies

MauriceP — Tue, 07 Dec 2021 22:35:04 GMT

This is not a questions so I am not sure if this is the correct place to post this? But I hope it is shareable and informative, it was fun anyway and I hope it prompts other users to contribute any similar experiments!

I suspect like other QDC users we have several dataflow constructs where output from dataflow A is input to dataflow B and so on...

An example of such a dependency tree is given below, where Octagons are QDC Sources, Rectangles Dataflows and Ellipses are Entities. The arrowheads show the direction of the data flow as either into a dataflow (LOADER) or output from the dataflow (STORE).

I am not aware of any QDC feature that can currently produce this inter-dataflow dependency (but I would be happy to be corrected!), so I built one - this was constructed in Python by creating a graph (DAG) where the Nodes and edges were derived directly from the QDC metadata database.

Possible uses for this information are:

documenting process flows
creating information for scheduling purposes
etc.

I have fully described the process in this github repository podium-dataflow-dependency-tree

Re: QDC Dataflow Dependencies

Joe_DosSantos — Fri, 17 May 2019 14:37:01 GMT

This is terrific Maurice. We had to use an external job scheduler for these dependencies at TD because we did not figure this out. I will make sure that the product management team sees this!

Re: QDC Dataflow Dependencies

MauriceP — Sat, 08 Jun 2019 18:01:59 GMT

Added support for node styles in the yaml config file. "Styles" control the node shapes, colours etc. The result is - I hope! -cleaner looking output.

Re: QDC Dataflow Dependencies

MauriceP — Sat, 08 Jun 2019 18:07:05 GMT

Given the output a cleanup and added support for node styles in the yaml.

Must show you my dataflow viewer sometime Joe - example below of our largest dataflow 115 objects (including sources and target)!

Re: QDC Dataflow Dependencies

Joe_DosSantos — Wed, 26 Jun 2019 14:16:04 GMT

Update for you. As part of the June Release, we will be adding a lineage dependency image that looks very similar to what you are looking for. I will be adding an update on the community page of the new features of QDC today, but wanted to alert you so that you can take a look at the post.