Build near real-time AI data pipelines with Spark Structured Streaming in Qlik Talend Studio
Fraud checks, in-session recommendations, and AI agents all need fresh data to make good decisions. Batch pipelines were built for yesterday's questions, computed on a schedule and handed over once a day.That gap matters more than ever: 88% of IT leaders now rank data streaming as a high investment priority, and Gartner lists agentic data streaming among its six top data and analytics trends for 2026, projecting adoption above 60% by 2028.
When you need streaming at scale, with complex joins and calculations across fast-moving events, that's where Spark's distributed processing earns its place over simpler ingestion tools. Support for Spark Structured Streaming, now in public preview inside Qlik Talend Studio, closes that gap: it computes on events as they arrive, using the same Spark skills and the same Spark clusters your team already runs.
Watch a short end-to-end walkthrough of a Structured Streaming job running in Talend Studio.
What's new
Near real-time processing in Talend Studio with your Spark clusters: Spark pipelines compute as events arrive instead of waiting for the next scheduled run.
Stronger reliability than the legacy Spark Streaming framework, with Catalyst optimizer support for better performance.
Joins across data speeds: enrich fast-moving events with slower-changing reference data, at scale.
Automatic time windows: events are grouped and aggregated over a moving period instead of being recomputed over fixed batch files.
Stateful joins across streams and clean recovery after a restart, handled for you. No streaming expertise required.
Runs on the Spark clusters: Databricks and AWS EMR.
At the same time that we are introducing Spark Structured Streaming, we are deprecating:
The legacy Spark Streaming framework, deprecated in Talend Studio from 8.0.1 R2026-09 onwards, with removal planned for R2027-09.
The following Built-In distributions on Spark Batch, deprecated in Talend Studio as of 8.0.1 R2026-09 with removal planned for R2027-09: Amazon EMR, Azure Synapse, Cloudera, Databricks, Hortonworks, Microsoft HD Insight 4.0, and Custom. The corresponding vendor platforms remain supported, but you'll need to use the Universal distribution instead of the Built-In one.
Support for HPE Ezmeral Runtime Enterprise through Kubernetes mode on Spark Batch, deprecated in Talend Studio from 8.0.1 R2026-09 onwards and scheduled for removal in R2027-09.
How to activate it
Spark Structured Streaming is available starting with Talend Studio 8.0.1 R2026-09 and later. It requires a Qlik Talend Cloud, Data Fabric, or Big Data license.
What to focus on when experimenting public preview
How your streaming sources behave (Kafka, Kinesis, or Pub/Sub) with Spark structured streaming jobs
Whether event-time windows and watermarking produce the metrics you'd expect when events arrive late or out of order.
How the pipeline recovers after a restart, and whether stateful joins across streams hold up under your workload.
How to share feedback
Try it, tell us how it feels, and help us shape the GA release. The fastest way to reach us is through your Qlik account team, or by replying to this thread directly. We're reading every comment. You can also submit a case through your usual support channel, with "Public preview" included in the subject.
