Qlik Community

Qlik Data Catalyst Discussions

Discussion Board for collaboration on Qlik Data Catalyst.

Highlighted
MauriceP
New Contributor II

QDC Dataflow Dependencies

This is not a questions so I am not sure if this is the correct place to post this? But I hope it is shareable and informative, it was fun anyway and I hope it prompts other users to contribute any similar experiments!

I suspect like other QDC users we have several dataflow constructs where output from dataflow A is input to dataflow B and so on...

An example of such a dependency tree is given below, where Octagons are QDC Sources, Rectangles Dataflows and Ellipses are Entities.  The arrowheads show the direction of the data flow as either into a dataflow (LOADER) or output from the dataflow (STORE).

 

prod_stg_compsych_member_t.png

 

I am not aware of any QDC feature that can currently produce this inter-dataflow dependency (but I would be happy to be corrected!), so I built one - this was constructed in Python by creating a graph (DAG) where the Nodes and edges were derived directly from the QDC metadata database.

Possible uses for this information are:

  • documenting process flows
  • creating information for scheduling purposes
  • etc.

I have fully described the process in this github repository podium-dataflow-dependency-tree

 

Tags (1)
Labels (1)
4 Replies
Employee
Employee

Re: QDC Dataflow Dependencies

This is terrific Maurice. We had to use an external job scheduler for these dependencies at TD because we did not figure this out. I will make sure that the product management team sees this!
MauriceP
New Contributor II

Re: QDC Dataflow Dependencies

Added support for node styles in the yaml config file. "Styles" control the node shapes, colours etc. The result is - I hope! -cleaner looking output.

 

prod_stg_compsych_member_t.png

MauriceP
New Contributor II

Re: QDC Dataflow Dependencies

Given the output a cleanup and added support for node styles in the yaml.

Must show you my dataflow viewer sometime Joe - example below of our largest dataflow 115 objects (including sources and target)!

Snag_135c748.png

Employee
Employee

Re: QDC Dataflow Dependencies

Update for you.  As part of the June Release, we will be adding a lineage dependency image that looks very similar to what you are looking for.  I will be adding an update on the community page of the new features of QDC today, but wanted to alert you so that you can take a look at the post.