Skip to main content
Announcements
See what Drew Clarke has to say about the Qlik Talend Cloud launch! READ THE BLOG
cancel
Showing results for 
Search instead for 
Did you mean: 
MauriceP
Contributor II
Contributor II

QDC Dataflow Dependencies

This is not a questions so I am not sure if this is the correct place to post this? But I hope it is shareable and informative, it was fun anyway and I hope it prompts other users to contribute any similar experiments!

I suspect like other QDC users we have several dataflow constructs where output from dataflow A is input to dataflow B and so on...

An example of such a dependency tree is given below, where Octagons are QDC Sources, Rectangles Dataflows and Ellipses are Entities.  The arrowheads show the direction of the data flow as either into a dataflow (LOADER) or output from the dataflow (STORE).

 

prod_stg_compsych_member_t.png

 

I am not aware of any QDC feature that can currently produce this inter-dataflow dependency (but I would be happy to be corrected!), so I built one - this was constructed in Python by creating a graph (DAG) where the Nodes and edges were derived directly from the QDC metadata database.

Possible uses for this information are:

  • documenting process flows
  • creating information for scheduling purposes
  • etc.

I have fully described the process in this github repository podium-dataflow-dependency-tree

 

Labels (1)
1 Solution

Accepted Solutions
MauriceP
Contributor II
Contributor II
Author

Added support for node styles in the yaml config file. "Styles" control the node shapes, colours etc. The result is - I hope! -cleaner looking output.

 

prod_stg_compsych_member_t.png

View solution in original post

4 Replies
Joe_DosSantos
Employee
Employee

This is terrific Maurice. We had to use an external job scheduler for these dependencies at TD because we did not figure this out. I will make sure that the product management team sees this!
MauriceP
Contributor II
Contributor II
Author

Added support for node styles in the yaml config file. "Styles" control the node shapes, colours etc. The result is - I hope! -cleaner looking output.

 

prod_stg_compsych_member_t.png

MauriceP
Contributor II
Contributor II
Author

Given the output a cleanup and added support for node styles in the yaml.

Must show you my dataflow viewer sometime Joe - example below of our largest dataflow 115 objects (including sources and target)!

Snag_135c748.png

Joe_DosSantos
Employee
Employee

Update for you.  As part of the June Release, we will be adding a lineage dependency image that looks very similar to what you are looking for.  I will be adding an update on the community page of the new features of QDC today, but wanted to alert you so that you can take a look at the post.