By reading the Product Innovation blog, you will learn about what's new across all of the products in our growing Qlik product portfolio.
A Quick Recap
We launched the Hybrid Data Delivery service in June 2021 with the aim of continuously providing enterprise data to your Qlik Cloud Analytics apps without you writing scripts or code. And I think we successfully achieved that goal. Enterprise data is streamed in two phases and is described in the diagram below.
On the left of the illustration are the enterprise data sources such as relational databases, SAP applications and mainframe systems with a Qlik Replicate appliance synchronizing data changes to a client-managed S3 bucket. This operation is controlled by Qlik Cloud Data Services by creating and running a “Landing Data Asset”.
The second step then uploads the data from the bucket into Qlik Cloud. Once again this is controlled by the Storage Data Asset you create within Qlik Cloud Data Services. The Storage Data Asset takes the data deltas from the S3 bucket, transforms them into a QVD format, then merges the changes into an existing QVD file, or creates a new QVD file if one doesn’t exist. One notable point to highlight is that all the data sets and assets are automatically cataloged and searchable in Qlik Cloud too!
Data Warehouse Data Gravity
OK, onto our big announcement… Fanfare, please. We’ve added Snowflake as a target to the Hybrid Data Delivery Service. Now you can continuously stream enterprise data updates to your Snowflake data warehouse as well as streaming data to Qlik Cloud with the same infrastructure. But let me address why folks want their data loaded into Snowflake before I dive into the expanded architecture.
Why Snowflake?
It should come as no surprise that business leaders now understand the importance of using data and analytics to accelerate digital business initiatives. Indeed, when COVID-19 hit, many organizations that used traditional analytics techniques that relied on vast swathes of historical data realized that they needed to pivot to gain agility. Consequently, many companies deployed cloud data warehouses to support agile analytics as a core business function. The traditional data warehouses were out and the cloud data platforms, such as Azure Synapse, Amazon Redshift, Google Big Query, and Snowflake were most definitely in. In fact, Snowflake was requested by prospects and customers alike and so that’s the warehouse we decided to support first.
The Expanded Architecture
You can see from the diagram below that our Snowflake support is a little different than our support of r Qlik Cloud and uses a different route to land the change data coming from enterprise systems. It’s still a two-step process (1. Landing 2. Merging and Storing ) however the raw data is first “landed” as tables within Snowflake which negates the need for an interim cloud file store. Once the data has been landed then the storage data asset fetches the changed data from the raw Snowflake tables and merges it with the regular Snowflake tables to complete the operation. You may also have noted that Snowflake is also supported on AWS, Azure and Google too.
What’s Next for the Hybrid Data Delivery Service
The final question that you’re probably asking is “What’s next for Hybrid Data Delivery?” I’m happy to describe a general direction but can’t go into specific timelines. Oh, and the legal disclaimer is that “all of this is subject to change.” Generally, we’re looking to support other data warehouses, such as Azure Synapse, Google Big Query, Amazon Redshift, and other targets like Databricks and streaming platforms too. We’re also considering replacing Qlik Replicate with a Qlik-managed data gateway. And finally, we’re investigating alternate data transformation services and rolling out additional data services. So, stay tuned for more innovations to come.
Conclusion
Companies are deploying cloud data warehouse infrastructure to facilitate many of their strategic analytics and data initiatives. Consequently, the Hybrid Data Delivery Service has been enhanced to help customers continually ingest enterprise data into their environment in near-real time without coding or scripting. With Qlik Cloud you can turn raw data into informed action – in one platform that shifts you from using a passive toolset to an active system built to deliver real-time information and drive immediate action.
Resources
Check out the following resources for more information about Qlik and Snowflake.