Automate your Machine Learning with Qlik Talend Cloud Data Pipelines

[damienedwards]

Use Qlik Talend Cloud to build a data pipeline that uses Machine Learning for real-time predictions to update data in your Qlik Sense application.

[damienedwards]

In today's data-driven world, organizations are increasingly leveraging Machine Learning (ML) to extract valuable insights from their data. This powerful technology enables businesses to make data-driven predictions, classify data, and uncover hidden patterns. As a result, ML can provide a significant competitive advantage, improve operational efficiency, and enhance customer experiences. 

However, implementing ML can be challenging. Two key obstacles often arise: poor data source quality and inefficient ML model development. Overcoming these problems requires providing accurate and complete source data for effective ML model training. Also, reducing the time-consuming and resource-intensive effort needed to develop and deploy these models. 

To overcome these challenges, organizations need a streamlined approach to integrate high-quality data with ML capabilities. Qlik simplifies this process, making it easier to build data pipelines that leverage the power of ML to drive better business outcomes. 

Introducing Qlik AutoML , Qlik Application Automation and Qlik Talend Cloud 

Qlik AutoML automates machine learning by using classification or regression models to find patterns in data that can be used for predictions. Qlik AutoML trains and tests your machine learning experiments, making them ready for deployment. These machine learning models can be integrated within Qlik Sense applications, Qlik Automation workflows and external applications.  

When configuring Qlik AutoML experiments, you select the target and features used within the predictive model.  Qlik AutoML automatically preprocesses trains and optimizes the model with the use of automatic feature engineering based on your choices.  Once the experiment is complete, your ML models can be deployed through APIs for real-time predictions. Qlik AutoML facilitates an iterative workflow by enabling you to tune your model parameters for better optimization.

Qlik Application Automation is a powerful tool that enables you to automate data, analytics and ML processes without writing any code. It provides a visual interface where you can easily create and manage automations, consisting of a sequence of actions and event triggers. Automations are a simple way to solve the data consumption use case for ML models where users can choose from templates or build their own workflows by assembling predefined connectors and logical blocks.

Qlik Talend Cloud 

Bringing it All Together 

Qlik Cloud capabilities allow you to use automation to create a workflow that acquires data from any supported source into a target with real-time predictions and load data visualizations within a dashboard application.  

We will demonstrate first using a  Titanic passenger survival data set to create and deploy a classification model utilizing AutoML. A data pipeline will be used to ingest and transform source data that can be used for predicting if a passenger would have survived the Titanic. An application automation will be created to invoke the classification model in real-time and update the resultant passenger survivor data in a QlikSense application. 

 The Qlik platform can show an example of providing predictions to a data integration pipeline with the use of automation. (Using the Titanic dataset from Kaggle, we can build a Qlik data pipeline that can predict which passengers survived the Titanic.) 

Setting up and running Qlik Talend Cloud Services 

Build a classification prediction Model using Qlik AutoML experiment using the Titanic data set. (Choose deployment model based on F1 score.) 

 
Deploy the generated CatBoost Classification model to predict survivors in our workflow using a Real-time prediction API URL.  

Qlik Talend Cloud Data Integration pipeline used to load source data from MySQL and transform the data for model predictions into a Snowflake target.  

Create a Qlik Automation to invoke the Qlik ML prediction model on the Titanic Transformed dataset created in the QTC Data pipeline. 

QLIK AUTOML CONNECTOR USING THE DEPLOYED TITANIC CLASSIFICATION API WITH FEATURES SHOWN BELOW 

Qlik Application Automation workflow sequence with embedded processor blocks. 

• Start Data Integration Pipeline data task to load dataset used for predictions. 

• The transformed dataset will call the deployed ML Flow Titanic Model API to generate classification prediction for passenger survival. 

• Load model predictions into Qlik Sense dashboard  

Qlik Sense Application Dashboard loaded with Real-time prediction data. 

 

Conclusion 

Qlik Talend Cloud delivers real-time prediction capabilities by adding machine learning to your data pipelines. The Qlik Application automation features make it easier to integrate the services Qlik Talend Cloud provides for data integration and analytics. The platform reduces the complexities of deploying ML models within your data pipeline and integrate the results for your Analytical application. Organizations can quickly adopt the power of machine learning within their enterprise data architecture with the Qlik Talend Cloud platform.