https://community.qlik.com/t5/Member-Articles/Tracing-data-lineage-for-SnowSQL-in-Talend-Data-Catalog/ta-p/2413603 <P>This article shows you how to configure two Talend Data Catalog bridges to harvest metadata from Snowflake and trace data lineage. It covers the <A href="https://help.qlik.com/talend/en-US/talend-data-catalog/8.1/Subsystems/Bridges/Content/MIRJdbcSnowflakeImport.htm" target="_blank" rel="noopener">Snowflake Database (via JDBC)</A> and <A href="https://help.qlik.com/talend/en-US/talend-data-catalog/8.1/Subsystems/Bridges/Content/MIRSqlScriptSnowflakeImport.htm" target="_blank" rel="noopener">Snowflake Database SQL DML (DI/ETL) Script (SnowSQL)</A> Talend Data Catalog bridges.</P> <H3><FONT color="#339966"><STRONG>Configuring the Snowflake database bridge import setup</STRONG></FONT></H3> <P>Configure the import setup parameters for harvesting metadata from a Snowflake database in Talend Data Catalog:</P> <OL> <LI> <P>Define the following parameters on the <STRONG>Import Setup</STRONG> tab: <STRONG>Host</STRONG>, <STRONG>Warehouse</STRONG>, <STRONG>Databases</STRONG>, and the <STRONG>User</STRONG> and <STRONG>Password</STRONG> for authentication.</P> </LI> <LI> <P><SPAN>Select the schemas to be harvested in the Schema field. This example uses the&nbsp;</SPAN><STRONG style="font-family: inherit;">TDC_TEST</STRONG><SPAN>, </SPAN><STRONG style="font-family: inherit;">TESTSCHEMA1</STRONG><SPAN>, and </SPAN><STRONG style="font-family: inherit;">TESTSCHEMA2&nbsp;</STRONG><SPAN>schemas.</SPAN></P> <DIV id="tinyMceEditorShicong_Hong_2" class="mceNonEditable lia-copypaste-placeholder">&nbsp;</DIV> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Shicong_Hong_3-1728870343274.png" style="width: 999px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172865iD324C48EAA355D14/image-size/large?v=v2&amp;px=999" role="button" title="Shicong_Hong_3-1728870343274.png" alt="Shicong_Hong_3-1728870343274.png" /></span></LI> <LI> <P>Go to the <STRONG>Overview </STRONG>tab to view an overview of the objects harvested from the Snowflake database.</P> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Shicong_Hong_4-1728870397512.png" style="width: 999px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172866iA45533491B8B95FD/image-size/large?v=v2&amp;px=999" role="button" title="Shicong_Hong_4-1728870397512.png" alt="Shicong_Hong_4-1728870397512.png" /></span></LI> </OL> <P>&nbsp;</P> <H3><STRONG><FONT color="#339966">Configuring the SnowSQL bridge import setup</FONT></STRONG></H3> <P>This example SnowSQL script contains a create view Data Manipulation Language (DML) statement:</P> <PRE class="ckeditor_codeblock">create view IF NOT EXISTS testschema2.v_employees ( employee_id, employee_name ) as select rvalue:employee_id::number, rvalue:employee_name::string from testschema1.employees; </PRE> <P>You will harvest metadata from this example SnowSQL script in the steps below.</P> <P>&nbsp;</P> <P>To configure the parameters for the Snowflake Database SQL DML (DI/ETL) Script (SnowSQL) bridge in Talend Data Catalog:</P> <P>&nbsp;</P> <OL> <LI> <P>Define the <STRONG>Directory</STRONG> where the SnowSQL file is located on the <STRONG>Import Setup</STRONG> tab.</P> </LI> <LI> <P>Add any necessary <STRONG>Include</STRONG> and <STRONG>Exclude </STRONG>filters to select the files to be harvested.</P> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Shicong_Hong_5-1728870568234.png" style="width: 999px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172867i7E35C49BBE04BE43/image-size/large?v=v2&amp;px=999" role="button" title="Shicong_Hong_5-1728870568234.png" alt="Shicong_Hong_5-1728870568234.png" /></span>&nbsp;</LI> <LI> <P>Go to the <STRONG>Overview</STRONG> tab to view the imported SnowSQL script.</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Shicong_Hong_6-1728870606064.png" style="width: 999px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172868i47AF840DEFC3BAEC/image-size/large?v=v2&amp;px=999" role="button" title="Shicong_Hong_6-1728870606064.png" alt="Shicong_Hong_6-1728870606064.png" /></span></P> </LI> <LI>View the connection defined for the SnowSQL model in the <STRONG>Configuration Manager.</STRONG><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Shicong_Hong_0-1728871701746.png" style="width: 999px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172877iA3BDADE83F0577DD/image-size/large?v=v2&amp;px=999" role="button" title="Shicong_Hong_0-1728871701746.png" alt="Shicong_Hong_0-1728871701746.png" /></span></LI> </OL> <P>&nbsp;</P> <H3><FONT color="#339966"><STRONG>Tracing data lineage for a field/column in the SnowSQL script</STRONG></FONT></H3> <OL> <LI> <P>Notice that after harvesting metadata (database objects and SnowSQL) using both bridges and building the configuration, the Architecture Diagram displays a bi-directional arrow, because the same Snowflake database model is the source and destination connection to the database.<span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="2024-10-14_10-00-37.png" style="width: 931px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172873iC78E94BAB51E8A1F/image-size/large?v=v2&amp;px=999" role="button" title="2024-10-14_10-00-37.png" alt="2024-10-14_10-00-37.png" /></span></P> </LI> <LI> <P>Go to the <STRONG>Data Flow</STRONG> tab to view the data flow diagram for the SnowSQL script. Referring to the SnowSQL script, Talend Data Catalog creates a view on one schema (<STRONG>TESTSCHEMA2</STRONG>) with a select statement on another schema (<STRONG>TESTSCHEMA1</STRONG>).<span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="2024-10-14_10-01-28.png" style="width: 999px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172874i36FF8EA2EE13541F/image-size/large?v=v2&amp;px=999" role="button" title="2024-10-14_10-01-28.png" alt="2024-10-14_10-01-28.png" /></span></P> </LI> <LI> <P>You can also view the data flow diagram for the database object (schema level), which displays the three schemas.<span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="2024-10-14_10-03-12.png" style="width: 999px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172875i82532DB4CA07A488/image-size/large?v=v2&amp;px=999" role="button" title="2024-10-14_10-03-12.png" alt="2024-10-14_10-03-12.png" /></span></P> </LI> <LI> <P>Finally, you can display the data lineage for a column (<STRONG>EMPLOYEE_NAME</STRONG>).</P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="2024-10-14_10-04-11.png" style="width: 999px;"><img src="https://community.qlik.com/t5/image/serverpage/image-id/172876i14CEAAFD7FE70F27/image-size/large?v=v2&amp;px=999" role="button" title="2024-10-14_10-04-11.png" alt="2024-10-14_10-04-11.png" /></span></P> </LI> </OL> <P>&nbsp;</P> <H4><FONT color="#339966"><STRONG>Environment</STRONG></FONT></H4> <UL> <LI>Talend Data Integration 8.0.1</LI> </UL> <P>&nbsp;</P> Mon, 14 Oct 2024 03:18:59 GMT Donna_Carr 2024-10-14T03:18:59Z https://community.qlik.com/t5/Member-Articles/Tracing-data-lineage-for-SnowSQL-in-Talend-Data-Catalog/ta-p/2413603 <P>This article provides a way to trace data lineage for a SnowSQL script. The scenario uses a Snowflake database table and Data Manipulation Language (DML) within a SnowSQL script.</P> Mon, 14 Oct 2024 03:18:59 GMT https://community.qlik.com/t5/Member-Articles/Tracing-data-lineage-for-SnowSQL-in-Talend-Data-Catalog/ta-p/2413603 Donna_Carr 2024-10-14T03:18:59Z