QDC Dataflow Dependencies

[MauriceP]

This is not a questions so I am not sure if this is the correct place to post this? But I hope it is shareable and informative, it was fun anyway and I hope it prompts other users to contribute any similar experiments!

I suspect like other QDC users we have several dataflow constructs where output from dataflow A is input to dataflow B and so on...

An example of such a dependency tree is given below, where Octagons are QDC Sources, Rectangles Dataflows and Ellipses are Entities.  The arrowheads show the direction of the data flow as either into a dataflow (LOADER) or output from the dataflow (STORE).

I am not aware of any QDC feature that can currently produce this inter-dataflow dependency (but I would be happy to be corrected!), so I built one - this was constructed in Python by creating a graph (DAG) where the Nodes and edges were derived directly from the QDC metadata database.

Possible uses for this information are:

• documenting process flows
• creating information for scheduling purposes
• etc.

I have fully described the process in this github repository podium-dataflow-dependency-tree

[Joe_DosSantos]

This is terrific Maurice. We had to use an external job scheduler for these dependencies at TD because we did not figure this out. I will make sure that the product management team sees this!

[MauriceP · Accepted solution]

Added support for node styles in the yaml config file. "Styles" control the node shapes, colours etc. The result is - I hope! -cleaner looking output.

[MauriceP]

Given the output a cleanup and added support for node styles in the yaml.

Must show you my dataflow viewer sometime Joe - example below of our largest dataflow 115 objects (including sources and target)!

[Joe_DosSantos]

Update for you.  As part of the June Release, we will be adding a lineage dependency image that looks very similar to what you are looking for.  I will be adding an update on the community page of the new features of QDC today, but wanted to alert you so that you can take a look at the post.