Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Most enterprise AI projects don’t fail because the model is wrong. They fail because the data isn’t ready. Data engineering leaders are now being asked to support a new wave of generative and agentic workloads that demand fresher data, broader source coverage, tighter governance, and richer context than traditional BI ever required — and to deliver it without growing the team.
Qlik Talend Cloud Data Integration was built to close that gap. It provides a single, governed pipeline from operational sources to an open lakehouse — and on to the vector indexes, feature stores, and APIs that your AI systems actually consume. Combined with Qlik Open Lakehouse on Apache Iceberg, it turns your AI inputs into reusable AI data products: named, versioned, governed assets that any RAG application or agent can consume off the shelf.
This post walks through the reference architecture, the pipeline that produces those data products, and a worked example that takes raw CRM and product data all the way to a working RAG copilot and an agentic workflow — both running off the same Iceberg foundation.
Why data is the bottleneck for enterprise AI
GenAI and agentic systems are not fundamentally different consumers of data, but they are far more demanding ones. A model is only as accurate, current, and trustworthy as the context it retrieves at inference time. For data engineering leaders, that translates into six hard requirements:
- Freshness — Embeddings and agent context become stale quickly. Real-time CDC matters more than nightly batch.
- Breadth — Useful AI requires content from CRMs, ticketing systems, document stores, ERPs, and operational databases — often dozens of sources per use case.
- Quality — Bad data doesn’t just produce bad answers. It produces confidently wrong answers, which are worse.
- Governance — PII, masking rules, lineage, and access controls must travel with the data into vector stores and tool calls, not stop at the warehouse boundary.
- Openness — Locking AI-ready data into a proprietary store creates rework every time the model, framework, or query engine changes.
- Reuse — Hand-rolling a new pipeline for every AI use case is how programs stall. The same curated data should serve a RAG copilot today and an agent tomorrow.
Meeting all six at once with one-off pipelines is what kills enterprise AI velocity. The path forward is consolidation: one governed integration platform feeding one open lakehouse, with the Gold zone publishing reusable AI data products that any model, agent, or analyst can consume. Build once, govern once, serve many.
Qlik Talend Cloud + Iceberg: a reference architecture
The architecture has four layers: sources, integration, an open Iceberg lakehouse with medallion zones, and an AI serving layer. Qlik Talend Cloud handles change data capture, transformation, quality, and catalog metadata across the entire flow. The Gold zone is where curated outputs are published as named AI data products.
|
Two design choices make this architecture work for AI specifically. First, the integration layer is real-time by default — log-based CDC keeps Bronze and Silver tables current without batch windows. Second, Gold is treated as a publishing surface, not a staging area. Each Gold data product is named, versioned, governed, and discoverable in the catalog. RAG and agents become two interfaces over the same products: built once, governed once, consumed many times. |
|
Figure 1. Reference architecture: Qlik Talend Cloud + open Iceberg lakehouse, serving RAG, agentic, and analytics workloads from the same governed Gold layer.
The pipeline: from raw data to AI use
The pipeline that operates on the architecture above runs in six stages — automated end-to-end, with quality and lineage enforced at every step. Each stage produces a more refined and trusted asset. Bronze preserves raw, append-only CDC for replay and audit. Silver applies data quality rules, deduplication, masking, and Type-2 history. Gold publishes AI data products: a document product (chunk-friendly text + metadata) for RAG, and a state product (curated entity, feature, and policy data) for agents. Both are versioned and registered, so consumers — vector indexers, semantic APIs, BI engines — read the same governed truth.
Figure 2. The six-stage pipeline. Because every stage writes to Iceberg, downstream consumers — vector indexers, semantic APIs, BI engines — read the same governed truth.
Worked example: from CRM tickets to a customer-support agent
Picture a data engineering team chartered with delivering an AI-powered customer-support assistant. The use case has both a RAG side (deflecting common questions with vetted answers) and an agentic side (the assistant can look up customer status, open tickets, and trigger actions). The raw inputs are typical:
- Salesforce — accounts, contacts, cases, case comments.
- ServiceNow — incident records and resolution notes.
- Confluence and SharePoint — a few thousand product KB articles.
- Postgres operational DB — subscription and entitlement state.
The pipeline at work
- Ingest. Qlik Talend Cloud uses log-based CDC to stream changes from Salesforce, ServiceNow, and Postgres in real time. KB articles are pulled on a connector schedule with content-hash detection so only changed docs flow through.
- Land in Bronze. Every change is written to append-only Iceberg tables in cloud object storage, partitioned by source and ingestion date. The raw audit trail is preserved for replay.
- Standardize in Silver. Push-down ELT cleanses text, masks PII (customer email, phone), conforms keys and status codes, and applies Type-2 history to entity tables (customer, case, entitlement, interaction). Trust scores are written alongside each table.
- Publish two Gold data products. rag_documents — KB articles + anonymized resolution notes from closed tickets, pre-joined and metadata-tagged for retrieval. agent_state — a fused customer_360 view, current entitlement state, and a small policy_rules table that defines what actions agents are allowed to take. Both are versioned, lineage-tracked, and registered in the catalog.
- Vectorize. rag_documents is chunked and embedded into a managed vector index with metadata filters for product, language, and access tier. The job is incremental — only new and changed rows of the data product trigger re-embedding.
- Serve and audit. agent_state is exposed via a thin Semantic API and parameterized SQL endpoints, ready for agent tool calls. Every agent action is written back to an audit_log Iceberg table — inputs, decision, tool call, outcome — so the same lakehouse that grounds the agent also makes its behavior explainable.
Powering RAG
When a customer asks “Why was my last bill higher than usual?”, the copilot retrieves the top-k chunks from the rag_documents data product, filtered by the customer’s product entitlement — with a structured lookup against agent_state for the customer’s current invoice context. Because the underlying data products are continuously refreshed by Qlik Talend Cloud, the copilot cites guidance that reflects the current pricing schedule, not last month’s. Every retrieved chunk carries its lineage, so answers can be traced back to a specific source row in Salesforce or a specific KB article version.
Powering agentic workflows
For agentic flows, the assistant plans and executes multi-step tasks against the same agent_state product: confirm identity, check entitlement, open a case in Salesforce via a write-back tool, and escalate to a human agent if confidence drops below a threshold defined in policy_rules. Every step is recorded in the audit_log table for explainability. The agent’s tools are backed by exactly the same data products the RAG side uses — which means a behavior change in the data, like a new product or pricing tier, propagates to both surfaces immediately, with no parallel pipelines and no copy-paste schemas. RAG and agents really are two interfaces over one set of products.
From pipeline to production: your next move
The fastest enterprise AI programs aren’t the ones with the cleverest prompts or the largest models. They’re the ones treating AI data products as the unit of delivery. Qlik Talend Cloud and Qlik Open Lakehouse give your team three things at once: real-time movement of broad source data, governed transformation into named and versioned data products, and an open Iceberg foundation that any model, framework, or agent can plug into. Build once, govern once, serve both RAG and agents from the same products.
A 10–15 day starting sprint for data engineering leaders:
- Pick one use case with two surfaces. Choose a domain where you need both a RAG copilot and a constrained agent (one or two write actions). Working backward from both surfaces forces the right data product shape.
- Stand up CDC into Iceberg. Wire two or three high-value sources into Bronze via Qlik Talend Cloud, build the Silver entity layer, and publish two Gold data products: one for retrieval, one for action.
- Measure freshness, trust, and reuse. Track event-to-context latency (freshness), quality-rule pass rate (trust), and how many AI surfaces consume the same Gold products (reuse). These three numbers tell you whether the pattern is ready to scale to the next domain — as configuration, not reinvention.
Talk to your Qlik team. Ask about the AI-ready data solution templates — pre-built pipeline patterns for the most common GenAI and agentic use cases, including the customer-service pattern walked through above.