Under the Hood of Qlik Cloud Data Services and Hybrid Data Delivery

[QlikProductUpdates]

I would be lying if I said that I wasn’t excited and a little nervous about our new release today because it’s been a while in the making. Months of planning and development have led to this point that truly marks a major event in both Qlik’s transformational journey to the cloud and for the next chapter in our data management strategy. Today our first data service is now generally available. WOOT! 

[QlikProductUpdates]

Qlik Cloud Data Services and Hybrid Data Delivery Summary 

Qlik’s vision is to deliver a whole set of cloud services that make it easier for you to work with data throughout your enterprise. The general term for a suite of related data services is EiPaaS, or Enterprise Integration Platform as a Service and Qlik Cloud Data Services is the umbrella name for Qlik’s EiPasS offering. Qlik’s EiPaaS is accessed from Qlik Cloud and uses many of the foundational services like catalog and Qlik Cloud Analytics. 

Hybrid Data Delivery is the first Qlik Cloud Data Services offering that is designed to replicate data in near real-time from on-premises source systems directly into Qlik Cloud. The service automatically ingests on-premises data into your tenant, catalogs the data and transforms it into a QVD format immediately ready for consumption by Qlik analytics without the need for job scheduling or scripting. What’s more, these QVD files are automatically and continually updated whenever the source data changes without any manual intervention. They are quite literally active. Hence, we call them ActiveQVD’s. Consequently, your analytics apps will always have the most up-to-date data for action and insight, whenever you need it. 

The Hybrid Data Delivery Architecture  

Suffice to say the new service borrows heavily from our market-leading, high-performance, change data capture technology with a bunch of new components running in Qlik Cloud. Once you’ve defined your data pipeline within Qlik Cloud and remotely started the delivery tasks, data is automatically and continuously delivered in QVD format into your Qlik tenant. A stylized representation of a running ingest pipeline is below: 

Now let’s peel the onion and dive a little deeper. The diagram below represents the major architectural elements of the new service. Our on-premises data sources are at the bottom of the picture: the databases, mainframes, enterprise apps etc. (currently the GA service supports relational sources only) and a server pre-installed with Qlik Replicate and Qlik Enterprise Manager. The Qlik Replicate server will “talk” to the data sources and the enterprise manager facilitates communication between Qlik Cloud and Qlik Replicate. The large grey box represents Qlik Cloud, with two white boxes: one represents the new data services and the other our analytics apps. The top left box represents an external cloud store, such as an AWS S3 bucket where raw change data is temporarily ‘’landed”. Solid lines represent data and metadata flow, while dashed lines represent control flows. 

Incidentally, our vision is to rework the need for the current Qlik Replicate / Enterprise Manager combo and replace it with “headless” data gateway technology. The gateway acts as a data broker only, with all command and control driven by the Qlik Cloud User Interface. Stay tuned for more details. 

Back to the diagram. Data flows clockwise and begins in Qlik Cloud with a new component called a Landing Asset. This correlates to a pre-defined, on-premises Qlik Replicate task that lands change data into an S3 bucket. Currently, the data service only allows you to remotely start and stop a task, (1) on the diagram. After the task is started, data is replicated to the landing zone whenever a change happens to the source database (2). 

(3) highlights the data flowing into another new Qlik Cloud component called a Storage Asset. The Storage Asset performs many steps because it’s responsible for continuously fetching data from the AWS S3 bucket, transforming the change data to QVD format, cataloging the fetched tables, and storing the result. There are currently two options for storing QVD files: 

1. Qlik managed inside Qlik Cloud.   

1. Customer-managed AWS S3 buckets. 

(My tip would be to stick with Qlik-managed for now 😊 ) 

Note: The Storage Asset synchronization process runs continuously but only applies the changes to the QVD in micro-batches after a change data partition is closed. The batch frequency is determined to the “configured interval” in the Qlik Replicate task. 

Since the QVD’s are continually updated with data, we call them Active QVD’s. Once the Storage Asset creates and catalogs the Active QVD’s, then they are ready to be consumed by your analytics applications. Eventually, the goal is to have the analytics apps automatically refresh too. 

I totally understand if this all sounds a bit complicated, and it would be far easier to see the solution in action. Well, today is your lucky day!  @Michael_Tarallo  has 2 great items for you. The first is a fantastic summary video and the second is a “Do More with Qlik - Webinar Recording” session dedicated to Hybrid Data Delivery. 

You can sign up for future Do More Session here: 

Click here for video transcripts

How Does Hybrid Data Delivery Compare to Qlik DataTransfer? 

I’ve summarized the various ways to load data into Qlik Sense SaaS in the diagram below: 

The goal of Qlik DataTransfer and Hybrid Data Delivery on the surface sound very similar, however, the use cases and scope are quite different. Qlik DataTransfer is a good utility for individuals to schedule data imports into Qlik Sense SaaS. But Hybrid Data Delivery is architected for the enterprise. The goal of Hybrid Data Delivery is to provide an enterprise-scale data movement solution that can import masses of on-premises data to Qlik Cloud, with future plans to stream data to other popular cloud platforms like Snowflake and Microsoft Azure Synapse. 

What Happens to Qlik Data Integration? 

Since I’ve just addressed Qlik DataTransfer, the other elephant in the room is to naturally ask about Qlik Data Integration. Firstly, Qlik Data Integration is not going anywhere. We’re adding new cloud services to complement our existing offerings. Our vision is to offer a Qlik hosted and managed EiPaaS (Enterprise Integration Platform as a Service). One that provides a single experience for all integration design patterns—including batch, streaming, CDC, ELT, ETL and a collection of data services (hybrid data delivery, transformation, application automation, data quality) to improve your analytics insights. 

Starting today Qlik Sense Enterprise SaaS customers will see a new “Data Services” icon in the switcher menu of your tenant which will take you to the new Qlik Cloud Data Services home (below). From here you can explore the new user interface, watch getting started videos and read the documentation. 

Conclusion  

Every company is either building or re-evaluating their data and analytics infrastructure, but many still struggle to incorporate data from their historical enterprise infrastructure and applications which leaves an informational void. The new hybrid data delivery service, part of Qlik Cloud Data Services, can fill that gap by automatically and continuously supplying on-premises enterprise data in near real-time to your business-critical Qlik analytics applications. 

Regards,
Clive Bearman
Product Marketing Manager