Direct Query now available in Qlik Sense SaaS!

[QlikProductUpdates]

Extend your analytics capabilities with cloud databases using Direct Query!  Qlik is excited to announce the release of Direct Query, a new capability that allows applications in Qlik Sense SaaS to directly query cloud databases using SQL pushdown as users interact with data through visualizations and user filtering.

[QlikProductUpdates]

Differing from our industry leading analytics engine, Direct Query generates SQL queries on the fly against the source and executes the query and compute at the database level. From big data to near real time use cases, Direct Query empowers users of any level to easily explore and analyze the most recent data right from the database, increasing the speed to insight and decision-making. 

Direct Query complements our best-in-class analytics engine and extends the range of consumption techniques for analyzing data from cloud databases. The first connector rolling out for Direct Query will be for Snowflake with Databricks next on the roadmap and others planned, so stay tuned.  

Can't see this video? - View it on the Qlik Video Page - Direct Query Part 1 - Overview and Operation

Can't see this video? - View it on the Qlik Video Page - Direct Query Part 2 - Building Hybrid Analytic Solutions

How does Direct Query work?  

Direct Query can be activated at the database connection level. So, using Snowflake as the example, when building a new application and starting with a data connection, you will have the option to use Direct Query versus our analytics engine.  

The user can then select certain components of the source database –in the Snowflake connector users will see the options for role, database, and schema. Authentication for Direct Query is governed at the database level so users will only have access to what is granted from the source database. 

The user can then select a single view or multiple tables with the option of choosing the type of relationship between the data sets like full outer join or inner join.  

Once the relation is designated, an import live statement is inserted into the script in the load editor on the back-end and the visualizations can begin being built on the front-end. Since this is a SQL pushdown tool, users will need to use the function library of the underlying database. As of this blog, we will support the major 5 aggregations of sum, avg, min, max, and count. Also, some simple forms of set analysis will be available for filtering within measures of charts. 

After the charts and objects have been built, users can filter and interact with data in real time as Direct Query generates SQL on the fly and pushes the query and compute down to the source database. Keep in mind that every selection or filter within the app from a user, results in a query pushed down to the database and compute. 

Direct Query will also feature a seamless transition into our analytics engine for advanced interactivity and the full suite of capabilities. Check out Part 2 ‘Building Hybrid Solutions’ from the videos below.  

To learn more, check out these two videos on Direct Query: 

Part 1: Overview and Operation 

Part 2: Building Hybrid Analytics Solutions with On Demand App Generation (ODAG) 

To learn more about how Qlik and Snowflake together can optimize your investment into data and analytics, read our corporate blog here.  

Join us for the webinar series, Do More with Qlik, on August 24th where our very talented Mike Tarallo will be showcasing Direct Query for Snowflake.  

[mehmet_gencsoy]

We need that feature urgently for Azure Databricks... do you have a timeline??

BR

Mehmet

[Michael_Tarallo]

Hi Mehmet - at this point I hear later part of 2022 - so please stay tuned. 

[pwagner]

For which 'usesases' do you recommend to use "direct query" and in which we should not use "direct query"?

Best regards, Patrick

[arychener]

Great!! Will this feature be available on-premise?

Best regards, Amélie

[luizcdepaula]

Original Question: "Great post! When will this be available for on prem as well?"

Let me rephrase my question. Currently, direct query is available in Qlik Sense on prem and we actually use it in a few apps against our SQL database. However, it is a little kludgy and performance doesn't seem to be great. Is there anything in this SaaS set up that helps with performance vs the On-Prem system?

Thanks

[AlexOmetis]

Direct Query in Qlik Sense on-prem is really quite different from Direct Query in Qlik Sense SaaS.

In Qlik Sense on-prem, the feature is actually called Direct Discovery even though the script command related to it is DIRECT QUERY. This allows a blend of in-memory data with SQL push-down for some other data. 

In SaaS, Direct Query a whole different way of building the data model for an app - the whole app is built with SQL push-down with no in-memory elements. Qlik Sense uses the defined relationships to write SQL that emulates the in-memory associative engine, without actually loading it into memory (other than some caching).

There's also Dynamic Views (which not too dissimilar from Direct Discovery in how it presents in the front end) and is available in both on-prem and SaaS. 

In terms of performance of any of these three, it's mostly tied to the source system as Qlik Sense has to wait for it to return the query results before showing them. I know there are some optimisations in Direct Query (SaaS) to only fetch the number of rows required for display as well as some caching, but fundamentally the performance will be dictated by Snowflake (or other sources in the future). 

[luizcdepaula]

Hi Alex,

Thanks for the detailed explanation. That is exactly what I was thinking, where the performance will be dictated by the source system. The main difference here though is the easiness of setting up the Direct Query in SaaS.

Thanks,

Luiz

[Patric_Nordstrom]

@pwagner Yes, Direct Query is a niche solution for some use cases, most of the time in-memory is faster, more flexible and much more cost effective. It’s good fit for browsing snowflakes databases and prototyping dashboards before loading into memory, or as a selection app for ODAG or showing aggregated KPI for fast moving databases for a small audience.

[pwagner]

@Patric_Nordstrom - thank you.

[BI41andAll]

So many data management systems have in-memory tech and can keep the data flowing quickly.

I welcome an in-memory cage match 😀

• "In this corner wearing the green trunks: QIX!
• In the opposite corner wearing dark pewter suspenders: Apache Arrow!
• In the 3rd corner wearing black & red fishnet: SQL Server Columnstore!"

Thanks for this feature.