Data Pipelines and Transformations

[Troy_Raney]

This Techspert Talks session covers:

• Creating pipelines in Qlik Talend Cloud Data Integration
• Demonstrating Transformation possibilities
• Preparing data for analytics

Chapters:

• 01:57 - Data Spaces
• 03:13 - How to create a connection
• 04:36 - How to create a project
• 07:50 - Landing task details
• 11:13 - Storage task details
• 13:29 - How to create transformation tasks
• 19:53 - Setting data for the catalog
• 20:50 - Data Quality settings
• 21:48 - Preparing data for analytics
• 22:22 - Using data from other
  22:59 - Triggering tasks with other tasks
• 24:11 - Q&A: How to use data from other projects?
• 24:51 - Q&A: How do you build data marts?
• 25:50 - Q&A: How to load data for analytic apps?
• 26:22 - Q&A: Is Talend Studio available in cloud?
• 26:59 - Q&A: Can this save in Iceberg tables?
• 27:35 - Q&A: Where to learn about loading from SQL?
• 28:05 - Q&A: What permissions are needed for new connections?
• 28:33 - Q&A: Can AI help build relationships in data?

Resources:

• Qlik Talend Cloud Data Integration Essentials
• Qlik Migration Center
• Setting up connections – Qlik Help Documentation

Q&A:

Q: Can we set notifications for changes in operation?

A: You can set these notifications at the task level to follow its execution:

Data task stopped due to an error / Data task status has changed to Stopped / Data task status has changed to Started / Table processing was suspended due to errors / Schema evolution during CDC for data task sources

Q: How to transform data directly after landing?

A: Some transformations are available in the storage task (which comes directly after the landing task): data rules and explicit transformations (row by row expressions, rename/add/remove columns, etc...)

Q: Why Does it Creates so many Tables?

A: Each task has its own schemas. The number of tables and views depends on the task configuration.

1.           Materialization (to create only views or views + tables)
2.          Historical data store (creates additional assets to store history of data changes)
3.          This structure allows to use Live Views.

The live views merge data from the internal schema tables with the latest data changes collected by the landing schema.

Therefore, live views always provide up-to-date data, even if the task has not yet run. In this case, data in live views is fresher than in tables.

Q: Where can we see the Errors faced in the task?

A: The task execution logs are available in the task monitor.

1.          You can create a Global monitoring view to check the statuses of the tasks of a same data space in a single console view.

Q: How to use that to understand more details about the error?

A: From the task monitor, click the three dots menu on the upper right corner and select View task logs.

From there, you can configure logging options to get more information by increasing the logs level (Error / Warning / Info / Trace / Verbose).

Q: Are these Data catalog options available to all types of Licenses?

A: Basic Cataloging is available in all Qlik Talend Cloud editions but advanced features such as Data Products, Data Marketplace and the Qlik Trust Score are only available in the Qlik Talend Cloud Enterprise edition.