Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Who is this For?
If you're a data engineer, platform architect, or on a DevOps team evaluating whether Qlik Talend Cloud's Dynamic Engine is the right move, or you're already committed and working through implementation, then this article is written for you. It covers architecture, configuration, and the operational patterns that matter in practice. Business value gets called out where it's relevant, but this isn't a sales pitch.
Kubernetes familiarity helps with some sections. It's not required to follow the concepts, but you'll get more out of the architecture details if you're comfortable with namespaces, operators, and Helm.
The Challenge with Always-On Engines
For most of Talend's history, data integration infrastructure followed a familiar and predictable pattern: provision a server or VM, install the Remote Engine runtime, and keep it running. The engine polls Talend Management Console (TMC) for work, processes jobs with limited concurrency, and runs whether it has tasks to execute or not.
For stable, predictable workloads that model is dependable. But for organizations dealing with variable demand, multi-tenant requirements, or teams trying to align their data infrastructure with modern DevOps practices, it creates real friction. You end up paying for compute you're not using, managing a fleet of VMs that need individual attention for upgrades, and hitting concurrency ceilings at exactly the wrong moment.
Dynamic Engine is Qlik's answer to those constraints.
What Dynamic Engine Is
The Dynamic Engine is a Kubernetes-native processing platform that executes Talend workloads like Data Integration jobs, Data Services, and Routes, on customer-controlled infrastructure. TMC and the still handle all management, scheduling, and orchestration. What changes is the execution layer. Instead of a persistent JVM waiting for work, Dynamic Engine provisions isolated containers on demand, runs the job, and releases those resources when it's done.
Three Principles Drive the Design
Execution is on-demand. There are no persistent runtimes to babysit. Each Talend Job, Data Service, or Route gets its own container for the duration of its execution. When the work is done, the container is gone.
Scalability is elastic. During peak processing windows, multiple containers run in parallel. During quiet periods, you're not burning budget on idle infrastructure. The ceiling is your cluster capacity, not a per-engine configuration value.
The design is cloud-agnostic and Dynamic Engine runs on Amazon EKS (including Auto Mode), Azure AKS, Google GKE (including Autopilot), and on-premises Kubernetes today, with Red Hat OpenShift support planned. Your cloud strategy stays yours.
What This Means in Practice:
- Cost: Eliminating idle resources can substantially reduce infrastructure costs compared to persistent engine models.
- Velocity: Developers spin up environments for testing without resource conflicts or provisioning delays.
- Reliability: Each job starts with a clean container, eliminating state-related failures common in long-running processes.
- Simplified operations: Infrastructure management is abstracted away, letting engineers focus on pipelines rather than execution environments.
Real-World Use Cases
Batch at scale: A retailer processing millions of nightly transactions scales from a handful of containers during the day to hundreds at peak, automatically.
Multi-tenant SaaS: A SaaS provider uses separate environments per customer, ensuring isolation while maintaining centralized monitoring, governance, and FinOps.
Hybrid cloud integration: An enterprise connects on-premises data warehouses to cloud analytics using environments configured with VPN access, maintaining security while leveraging cloud scalability.
Architecture: Two Environments, One Control Plane
Dynamic Engine spans two distinct environments: the Qlik Talend Cloud layer managed by Qlik, and your Kubernetes cluster where execution happens. They communicate over HTTPS, with your cluster always initiating the outbound connection. No inbound firewall rules are required on your cluster for Dynamic Engine.
On the Qlik side, TMC is the user-facing control plane. Through it you define Dynamic Engine entities, manage Dynamic Engine Environments (DEEs), configure Run Profiles, deploy and monitor tasks, and trigger promotions across environments. The Talend Cloud API exposes the same operations programmatically via REST, and it's worth noting this is the same API available for all Talend Cloud interactions and not something specific to Dynamic Engine.
Inside your cluster, two namespaces do the work.
The Two-Namespace Design: Stability Meets Elasticity
This is one of the more important architectural details to understand, because it explains how Dynamic Engine achieves two things that typically work against each other: a stable, persistent control layer and an execution layer that scales up and down freely.
The infrastructure namespace is the fixed foundation. It contains the engine-operator, which is the Kubernetes operator responsible for managing the lifecycle of all Dynamic Engine resources in the cluster, along with the CRDs that extend the Kubernetes API with Dynamic Engine-specific resource types, and the associated RBAC primitives, so the ClusterRoles, RoleBindings, ServiceAccounts, Secrets, and ConfigMaps. This namespace is deployed once per Dynamic Engine instance and its footprint doesn't change with workload.
It also hosts a local container image registry used for Data Service and Route images. When a Data Service or Route task is deployed, an image-builder job builds the task-specific container image and pushes it to this registry, where execution pods pull from it. Batch Data Integration jobs use a different artifact delivery path.
If your organization requires internally managed and secured registries, your own container image registry can be substituted for the in-cluster repository.
The Dynamic Engine Environment namespace is where execution actually happens. Each DEE you create in TMC maps to its own namespace in the cluster, named qlik-processing-env-<env-id> by default, though this is customizable via Helm. This namespace contains the service pods that enable task execution, the ephemeral job pods running your workloads, and the persistent storage resources supporting them.
One Dynamic Engine instance can serve multiple Environment namespaces simultaneously. That one-to-many relationship is what makes it practical to maintain separate dev, test, and UAT environments, or to organize by business unit, like accounting-dev, marketing-uat, all without needing separate clusters or engine deployments.
How the Two Layers Communicate
Two channels connect Qlik Talend Cloud to your cluster.
HTTPS handles provisioning and management. The engine-operator establishes an outbound connection to Talend Cloud services, through which the TMC issues deployment instructions, configuration changes, and version upgrades. All traffic is TLS-encrypted.
ActiveMQ over HTTPS handles Talend job dispatch. Environment namespace service pods use this channel to receive task execution requests and report status back to TMC. The reliable message delivery model means a job submission isn't lost if your cluster is temporarily unreachable. Like the HTTPS channel, this is outbound from your cluster, so no special inbound firewall rules needed.
Air-gapped and Restricted-Network Deployments
Dynamic Engine has been designed with security and risk mitigation in mind. Both channels support air-gapped deployments. Infrastructure Helm charts and required images can be sourced from a local registry, such as Harbor, though cloud provider registries and other commercially supported options are used as well. In this setup, your cluster nodes never pull images from the Internet.
In this example Helm command, you’ll see the domain references the internal registry.
helm install dynamic-engine-crd \
oci://your.internal.registry/talend/helm/dynamic-engine-crd
For TMC communication in restricted environments, an HTTPS proxy can be placed between your cluster and Qlik Talend Cloud services. You can find additional information for configuring the proxy in your Dynamic Engine setup in the documentation.
Dynamic Engine also provides support for Kubernetes Network Policies, allowing you to enable default network policies that are set during your installation or upgrade, and further, you're able to further define custom egress rules that restrict your outbound traffic.
Deploying Dynamic Engine
Once you've created a Dynamic Engine in TMC, deploying it to your cluster uses familiar Helm install, and it exposes the full range of configuration values, supports helm upgrade with rollback capability, and lets you validate configurations before applying with --dry-run=server.
Prerequisites include needing a cluster to be running Kubernetes v1.30–v1.36, and you'll need cluster admin permissions for CRD registration and namespace creation.
One thing worth highlighting is that once Dynamic Engine and its Environments are up and running, data engineers don't need cluster access for day-to-day work. All task deployment, monitoring, and management happens through TMC and the Talend Cloud API. Direct cluster interaction is only needed when adding, removing, or upgrading Dynamic Engine components themselves.
Customizing the Execution Environment
Dynamic Engine environments support a range of customization through standard Kubernetes resource mounting. ConfigMaps work well for non-sensitive configuration like API endpoints or feature flags. Secrets can be injected as environment variables for credentials, or mounted as volumes when jobs need access to certificates or keystores. PersistentVolumeClaims cover shared storage scenarios where data needs to survive beyond a single pod's lifetime. These patterns can be combined, and they apply independently to Data Integration job pods and Data Service or Route pods, so you have separate control over each execution type.
Run Profiles give data engineers a TMC-native way to tune how jobs run without touching cluster configuration directly. A Run Profile defines the JVM arguments and Kubernetes resource parameters for a given task: CPU and memory requests and limits, replica counts for Data Services and Routes, and autoscaling thresholds if needed. A team running a memory-intensive transformation job can have its own profile with higher pod limits, while lighter workloads share a more constrained default.
Because Run Profiles are managed in TMC, data engineers can adjust execution parameters within the boundaries the platform team has established, without needing cluster access.
Talend Run Profiles defined in Talend Management Console allow you to define resource configuration, JVM arguments, and Kubernetes Pod resources, allowing for finer declaration of your configuration.
Upgrades Without the Usual Pain
Upgrade management is one of the clearest operational advantages Dynamic Engine has over the Remote Engine model, and it's worth walking through how it works.
When a new version is available, TMC surfaces the notification directly in the console. Applying it uses the same manifest or Helm commands as the original deployment. The critical difference from Remote Engine is that running jobs and tasks are not interrupted. Kubernetes' rolling update mechanism brings up new engine pods, validates them, and only then terminates the old ones. The engine's control capabilities remain available throughout.
If something goes wrong during an upgrade, such as a misconfigured values file, an incompatible cluster state, an image pull error, the engine will roll back to the previous working configuration automatically. There's no manual recovery process and no separate rollback procedure to maintain.
The same rolling update behavior applies to your own Data Services and Routes. Deploying an updated version through TMC uses Kubernetes' RollingUpdate strategy, keeping your application available as the new version comes up and gets exposed.
Monitoring What Matters
TMC is the primary operational window into running workloads. For batch Data Integration jobs, the signals that matter are execution status, duration, and log output. For always-on Data Services and Routes, the primary health indicator is replica status and the desired replica count versus those currently running, which tells you immediately whether your services are healthy.
Fitting Into Your Kubernetes Ecosystem
One of the practical advantages of a Kubernetes-native architecture is that Dynamic Engine and its Environments slot into the tooling your team already uses. Standard Kubernetes-compatible monitoring, logging, telemetry, and egress solutions work without customization. You're not adopting a proprietary observability stack or working around a closed runtime, you're running containers in namespaces, and that means Prometheus, Grafana, Fluentd, your GatewayAPI of choice, and whatever else your platform team has standardized on, all apply.
When Dynamic Engine Is the Right Call
The architectural differences between Remote Engine and Dynamic Engine aren't superficial. Across nearly every dimension like runtime model, concurrency, cost, and job isolation, the two runtime engine types reflect fundamentally different assumptions about how data integration infrastructure should work.
Core Architectural Differences
| Feature | Remote Engine | Dynamic Engine |
|---|---|---|
| Runtime Model | Persistent OSGi container, long-running JVM | Kubernetes-native; ephemeral pods/execution |
| Concurrency | 3 parallel tasks (default, configurable per engine) | Bound by cluster capacity; no per-engine cap |
| Resource Model | Always-on; cost is continuous regardless of use | On-demand; cost scales with execution time |
| Job Isolation | Shared JVM for concurrent tasks | Per-pod isolation; namespace per execution |
| Infrastructure | Customer VMs (EC2, Azure VM, on-prem) | Customer Kubernetes (EKS, AKS, GKE, etc.) |
| Upgrades | Manual installation per engine VM | TMC-notified; rolling update, rollback on failure |
| Impact to Running Jobs | Engine restart affects running jobs | Engine upgrades do not affect running jobs |
| OSGi Artifacts | Fully supported | Not recommended; use Route or Data Service as microservice deployment type |
| Routes / Data Services | Supported (OSGi model) | Supported (container-image model) |
Those differences translate directly into the workload patterns where Dynamic Engine has a clear advantage.
Variable demand workloads where nightly batch spikes, end-of-month processing, and event-driven integration, for example, benefit directly from on-demand compute. You stop paying for headroom that you only need occasionally.
Multi-tenant and regulated workloads get genuine namespace-level isolation with Kubernetes RBAC, without the overhead of separate clusters per tenant.
If Kubernetes is already part of your infrastructure, the marginal overhead of adding Dynamic Engine is low. The operational model is already familiar.
When concurrent job volume outgrows what a Remote Engine fleet can reasonably handle, Dynamic Engine's use of Kubernetes' Horizontal Pod Autoscaler is the natural path forward.
And for teams spending meaningful engineering time maintaining execution agents and managing engine upgrades, the TMC-managed upgrade model and automatic rollback free that time up for work that actually moves the needle.
Where This Is Heading
Dynamic Engine reflects a broader direction the data integration industry is moving: toward serverless, consumption-based models where infrastructure costs track actual usage rather than theoretical peak capacity. As organizations continue to mature in their cloud adoption, that kind of flexibility stops being a differentiator and becomes a baseline expectation.
Qlik Talend Cloud's Dynamic Engine gets you there without trading away the reliability and feature depth that enterprise data teams depend on. The architecture is sound, the operational model is genuinely simpler than what it replaces, and it fits into the Kubernetes-native infrastructure patterns most platform teams are already building toward.
Practical Guidance for Getting Started
A few design and deployment patterns that tend to matter in practice:
Start with modest resource configurations and scale based on actual metrics. Over-provisioning environments is easy and expensive, and that's one of the things Dynamic Engine is designed to help you avoid.
Keep environments separated by purpose. Dev, UAT, and production should be distinct DEEs. The namespace isolation is there, so use it to prevent accidental cross-environment interference.
Design Talend jobs for parallelism where it matters. Stateless job design, or managing state externally to ephemeral container storage, lets you fully exploit Dynamic Engine's ability to run multiple instances simultaneously.
Use TMC's monitoring to track execution times, resource utilization, and cost over time, and treat that data as feedback for configuration refinement. The consumption-based cost model rewards this kind of attention.
Wrapping Up
The per-pod isolation and environment-level namespace separation add up to something concrete. Execution environments stay clean and isolated without the overhead of separate clusters per tenant or business unit, and infrastructure absorbs variable demand without manual intervention. For organizations managing multi-tenant workloads or a growing portfolio of data integration pipelines, those properties translate directly into fewer incidents caused by shared runtime state and more predictable operations at scale.
The technical foundation makes that possible without trading away operational control. Your cluster, your security policies, your observability stack. Dynamic Engine fits into the infrastructure you're already running rather than replacing it.