Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
As the needs for data management evolve rapidly and the demand for large-scale processing increases, Qlik takes a bold step forward with the release of its groundbreaking product: Dynamic Engine.
This processing engine, designed to integrate natively with Kubernetes (K8s), redefines the data processing architecture, offering a unified, scalable, and future-ready solution.
In this blog, we will explore the key features of Dynamic Engine and what sets it apart from existing processing solutions within the Qlik Talend Data Fabric.
Unifying and Scaling Data Processing at Large
Dynamic Engine introduces a unified processing platform that adapts seamlessly to any workload, whether deployed in on-premise, hybrid, or SaaS environments. This flexibility makes it the solution of choice for enterprises looking to migrate or manage their data pipelines across diverse infrastructure setups.
Here are the key advantages of Dynamic Engine:
- Natively Integrated with Kubernetes (K8s): By leveraging Kubernetes, Dynamic Engine processes jobs within pods, offering nearly unlimited horizontal scalability. Each project is isolated within its own environment (namespace-based), ensuring better resource management and enhanced security.
- Distributed Processing at Scale: Unlike traditional Talend engines, Dynamic Engine enables the deployment of Kubernetes clusters to execute tasks in parallel, maximizing performance for batch processing and real-time data flows.
- Seamless Migration from Remote Engines: For existing Remote Engine users, the migration to Dynamic Engine is designed to be smooth and painless. Dynamic Engine becomes the new standard, ensuring a seamless transition while benefiting from enhanced features.
- Unified Data Workflows: One of Dynamic Engine’s main goals is to unify data processing workflows across the entire Qlik Talend suite, whether it’s integration, transformation, or data quality management. The engine standardizes and automates these workflows, regardless of data sources or destinations.
Dynamic Engine enables businesses to orchestrate data integration tasks on customer-controlled infrastructure while benefiting from cloud-managed services. But what sets it apart from traditional engines like Talend’s Remote Engine?
Built-in Version Upgrade Mechanism
Dynamic Engine simplifies the process of keeping your environment up to date with the latest versions. With a built-in version upgrade mechanism, users will receive notifications via the TMC whenever an update is available. The versioning system ensures consistency between the Dynamic Engine and its related Dynamic Engine Environments, which makes managing upgrades across different environments a straightforward process.
This mechanism allows for easier updates, ensuring that users can always benefit from new features, security patches, and performance improvements without the manual effort typically required in version management.
Smooth Migration from Remote Engine
For those currently using Qlik Talend’s Remote Engine, the transition to Dynamic Engine is made seamless through TMC’s promotion-based migration path. This migration is designed to be as frictionless as possible, leveraging existing APIs and known workflows.
With a few steps, users can promote their existing Remote Engine setups to Dynamic Engine configurations, preserving the familiarity of the existing environment while taking advantage of the added flexibility and cloud-native capabilities of Dynamic Engine.
Scalability via Run Profiles
Dynamic Engine’s ability to scale dynamically is one of its strongest features. Using TMC’s Run Profiles, organizations can define how their data tasks are distributed across resources.
This level of customization provides businesses with the flexibility to optimize their resources, improve performance, and reduce costs—all directly managed through TMC.
Compatibility with Leading Cloud Providers and On-Prem Infrastructure
Dynamic Engine is designed to work across various cloud and on-prem infrastructures, making it a versatile choice for enterprises. It is currently compatible with:
- AWS EKS (Elastic Kubernetes Service)
- Azure AKS (Azure Kubernetes Service)
- On-premise environments like RKE2 and K3S
In the near future, compatibility will extend to Google GKE (Google Kubernetes Engine) and OpenShift, ensuring that Dynamic Engine can meet the needs of organizations across different platforms. This flexibility allows businesses to maintain a hybrid approach to cloud and on-prem infrastructure, aligning with their specific requirements.
Comparison with Remote Engines
Historically, Talend’s solutions relied on Remote Engines to execute jobs outside of the Cloud. These engines allowed enterprises to maintain control over their infrastructure while utilizing local processing capabilities. However, as scalability and flexibility demands grew, these engines faced some limitations. Dynamic Engine, on the other hand, positions itself as a modern and automated solution.
| Remote Engine | Dynamic Engine |
|
Limited Scalability: Remote Engines were constrained by the capacity of the machines they ran on. For large workloads, the infrastructure had to be manually adjusted constantly. |
Automatic scalability powered by Kubernetes. |
|
Fragmented Data Flows: Although highly performant, Remote Engines required specific configurations for each type of processing, leading to fragmented workflows. |
More fluid orchestration of data flows, centrally controlled via the Talend Management Console (TMC). |
|
Manual Environment Management: Each Remote Engine required a high level of manual management for scaling and resource optimization. |
Optimized resource utilization through Kubernetes pods, which can be easily provisioned and managed dynamically. |
How It Operates and Deploys Across Different Environments
Dynamic Engine is designed to be deployed seamlessly in various environments. Here’s a closer look at its operation:
- On-Premise (with Cloud Orchestration via TMC): Although Dynamic Engine can be deployed on a Kubernetes cluster located on an on-premise infrastructure, its management and orchestration are still handled through the Talend Management Console (TMC) in the Cloud. This means that enterprises with local infrastructures can leverage their existing resources while benefiting from Cloud flexibility and agility for job management, performance monitoring, and workload scaling. This hybrid approach allows businesses to retain control over their physical resources while centralizing environment management through the Cloud.
- Hybrid or SaaS (via Kube as a Service approach): For enterprises operating in hybrid or fully Cloud-based environments, Dynamic Engine integrates seamlessly via platforms such as Qlik Cloud or directly through TMC. This flexibility allows Kubernetes clusters to be deployed in the Cloud and jobs to be managed at scale without requiring local infrastructure, making deployment simpler and faster.
- Scalability through Kubernetes: One of Dynamic Engine’s major strengths is its ability to dynamically provision Kubernetes nodes based on processing needs. This ensures that data pipelines can automatically scale with workload variations, maximizing resource efficiency and reducing processing times.
Full documentation can be found here.
Conclusion: A Future-Ready Engine
With Dynamic Engine, Qlik offers a solution that not only addresses today’s challenges of large-scale data processing but also sets the standard for future data management needs. Whether enterprises are looking to scale their processing capacity, unify their data workflows, or automate environment management, Dynamic Engine stands out as the solution of choice.
Together with Talend Data Fabric, Dynamic Engine creates a complete ecosystem that transforms how data is integrated, processed, and leveraged across the organization.