STT - Exploring Qlik Cloud Data Integration

[Troy_Raney]

This Techspert Talks session covers:

• What Qlik Cloud Data Integrations can do
• How it can be used
• Cloud Data Architecture

Chapters:

• 01:10 - Data Integrations Architecture
• 03:35 - Look at a complete Data Project
• 06:37 - Building a new Data Project
• 12:05 - Type 2, Data History
• 14:57 - Task monitoring
• 17:02 - Space access control
• 18:10 - Prep for Analytics layer
• 19:01 – Transformations
• 21:23 - Q&A: How many connections are free?
• 22:15 - Q&A: Do the sources need to be cloud-based?
• 23:15 - Q&A: What cloud security is in place?
• 24:36 - Q&A: Transformations and DBT in Qlik Cloud?
• 26:03 - Q&A: When exactly does encryption occur?
• 27:10 - Q&A: Possible to connect to QVD files?
• 27:53 - Q&A: Can Section Access be used on Data Pipelines?
• 28:27 - Q&A: Is ODBC driver needed with Snowflake?
• 29:12 - Q&A: Are there Data Mart courses?
• 29:43 - Q&A: Difference between Talend and Stitch?

Resources:

• Trust and Security at Qlik
• Troubleshooting Qlik Data Gateway Direct Access
• Exploring Qlik Application Automation
• Using Qlik Cloud Data Integration to Onboard and Transform Data

Q&A:

Q: Does QCDI works as orchestration?

A: QCDI has built in orchestration capabilities for its tasks.   This includes scheduling features and workflow.

Additionally, Qlik Cloud provides Qlik Application Automation which can handle broader orchestration capabilities including Qlik Cloud components like QCDI or external components.  For example with Qlik Application Automation, you could run QCDI tasks to load and transform data, then execute a databricks notebook or job and then run your Qlik Sense reload.

 Q: Does data flow through the QCDI cloud region or does my gateway region matter for it?

A: For data movement it depends on the data source.

Relational databases, mainframe and SAP (ECC / S4/HANA) data is captured by the gateway and delivered directly to the target without passing through Qlik Cloud.  

SaaS applications like ServiceNow, Salesforce etc. do pass through our cloud.

Q: Do you also have Rest-API to Jira data?

A: There is a connection available to Jira Cloud in QCDI.

Q: Are Talend integration and Qlik Cloud Data Integration the same?

A: No they are not the same.  

Qlik acquired Talend and we are working to integrate features from Talend Cloud into Qlik Cloud – come to Qlik Connect to learn more!

The use cases solved by Talend and Qlik Cloud Data Integration are different also….

QCDI is focused on 2 core use cases:

• Data Replication

Log-based, low-latency capture for RDBMS sources and delivery to RDBMS or cloud object store targets Incremental (API based) capture for SaaS sources and deliver to RDBMS or cloud object store targets

• Data Pipelines for Data warehouse and data lake houses

Supporting Snowflake, Databricks, Google BigQuery, MS Fabric, AWS Redshift and SQL Server(onpremise or cloud)

This provides data ingestion AND transformation capabilities.

Transformations are performed using an ELT paradigm where the workload is pushed down to the data platform.

• Talend Data Fabric w/ Talend studio is a data integration solution that solves a wide variety of use cases

ETL process with Java or spark based transformations that leverage a mid-tier engine to process data (not ELT).

Application integration workloads that can integrate applications together

Streaming or event based integrations with message queues / kafka or other eventing systems

API creation with no-code features to define API’s

At the end of the day you can combine capabilities but you should consider your architecture / use case to determine the best solution.   Talend Data Fabric / Studio is Qlik’s go-forward, strategic ETL solution while QCDI is our go-forward, strategic ELT solution

Click here to see video transcript