topic SV:Re: Statistical and predictive modeling in QlikView

Statistical and predictive modeling

Tue, 17 Aug 2010 01:44:46 GMT

I have a few questions about how people are doing statistical analysis/predictive modeling/data mining with QlikView. According to the Gartner 2010 BI Quadrant, QlikView "lacks the statistical and predictive modeling capabilities of some of its most similar competitors, ..."

My questions:

1) How often are you asked to do predictive modeling with QlikView? (whether you're an in-house developer or work for a consulting firm)

2) What is the most advanced statistical work you've done inside of QlikView?

3) Does anyone use third party statistical packages to pre-calculate before loading into QlikView? (R anyone?)

Thanks,

Gary

Statistical and predictive modeling

johnw — Tue, 17 Aug 2010 22:46:57 GMT

Well, this won't be very helpful, but it's AN answer.

1) Never. In house developer.
2) Basic statistical distribution stuff, box plots, 95th percentile, standard deviation, that sort of thing.
3) Not me.

SV:Re: Statistical and predictive modeling

larsc — Wed, 18 Aug 2010 00:19:43 GMT

I did some laboration with the rangecorrel function a while ago, but never got the chance to finalize anything.

It could be useful to display the correlation between two (or more) expressions in some scenarios. Let's say number of customers and number of orders.

If the correlation factor between these two expressions is low, the increased amount of orders is probably not a result of the increasing amount of customers, it is probably related to something else.

It may be possible to take this further by analyzing the correlation between more than two expressions, to find the reason behind a trend change in a selected expression.

It is always useful to show a customer the cause behind a trend change in different ways, not only that a trend change exists.

You may play with the rangecorrel function using a line chart with these three expressions:

1) Count(CustomerID)
2) Count(SalesOrderID)
3) rangecorrel(above(total Count(CustomerID), 0, 100),above(total Count (SalesOrderID), 0, 100))

It would be very nice if QlikTech (or someone else) could publish some nice demo's of this, and other statistical functions.

SV:Re: Statistical and predictive modeling

Thu, 19 Aug 2010 00:32:27 GMT

John and Lars,

This is exactly what I was looking for, including lack of usage. I am guessing that a majority of QlikView use cases don't include any predictive modeling. 95%? 99%?

I would also like it if QlikTech could promote use cases of predictive modeling within QlikView. Some of our clients are starting to ask about its features and I am investigating the best ways to incorporate it (or not).

Thanks,
Gary

SV:Re: Statistical and predictive modeling

johnw — Thu, 19 Aug 2010 00:44:53 GMT


Gary Strader wrote:I am guessing that a majority of QlikView use cases don't include any predictive modeling. 95%? 99%?

99% or higher wouldn't surprise me. Predictive modeling seems likely to be one of the harder things you could do in QlikView. But I see almost no posting on the subject on the forum by people asking for help. So it seems reasonable to conclude that very few people are doing it. Lack of posts on the subject aren't proof, of course, but it's suggestive.

On the other hand, if it IS 99% or higher, it is also strange that you have more than one client asking about it. Guess you're just lucky. 🙂

SV:Re: Statistical and predictive modeling

Thu, 19 Aug 2010 01:03:19 GMT

I think you're right about the lack of posts. There are only a handful returned on a forum search.

What's interesting is that while many clients are asking about it, none have actually asked for us to implement it. It is more when they are trying to evaluate whether to purchase QlikView versus the competitor products, they want to know if it has this or that feature. And predictive modeling keeps coming up.

Statistical and predictive modeling

Thu, 19 Aug 2010 01:04:05 GMT

I've had a few people ask ABOUT it, (in terms of assessing our capabilities in RFP; we are a QlikView OEM partner) but I have yet to have anyone actually request to implement anything using the tool. The closest I've come is estimating monthly totals based on current daily averages… which I would hardly call predictive analysis. I've also done some 'what-if' type analysis, but again, nothing overly predictive.

Most of the customers that I've implemented it for are blown away by the level of access and insight they gain from QlikView. I think they're overwhelmed by knowing where they've been, or where they've come from to even worry about our ability to predict where they're going, beyond following a trend line.

'Predictive Analysis' is one of those things that people have learned to ask for, but generally have no idea what it is much less how to use it. (Voila! A bicycle for your fish!)

SV:Re: Statistical and predictive modeling

larsc — Thu, 19 Aug 2010 08:06:30 GMT

I have actually yet not had any request at all, from any customer that made me have to use any of the statistical functions. But that doesn't mean they aren't powerful, and can't contribute to the QlikView experience. I'm sure they can!

For example, such a thing as finding the most probable cause behind a trend change using the rangecorrel function. There must be many areas where this can be useful, implemented in a "user friendly" way. The problem is, using these functions might need some explanation depending of the end users. Not everyone knows what a correlation coefficient is

Maybe all we can do is to start a group in this forum, where we who are interested in the statistical functions can start to develop and share some stuff together!

SV:Re: Statistical and predictive modeling

Thu, 11 Nov 2010 16:05:55 GMT

Hi,

We deal with call volumes and most of the modeling is done through another application, which we tap into and pull finished results via the View made available. However, we have on occasions tried to make some predictive figures based upon historical events and took the mean figures to apply into the future. It's not exactly Erlang, but in a short period it can make fairly accurate projections. Our big buzz just now is using Distribution curves and I have felt Qlikview as a tool fails to deliver thus far. This maybe our building that is the failure, but either way we have not found Qlikview the reliable tool in this area to date.

I would be happy to see if anyone has better success as we use Qlikview heavily for repeatable reports and it's likely useful for much more.

Thanks

Peter

Statistical and predictive modeling

danielrozental — Fri, 12 Nov 2010 01:17:17 GMT

1) Very rarely. Consultant

2) Calculating a linear regression, a colleague did a Beta function calculation.

3) I've used R to calculate a multiple linear regression, but it was just for a prototype. Batch offline integration should be quite possible using command line instructions, not so sure about interactive analysis though.

Will definitely love QT to add more features regarding this but really doubt they will.

Statistical and predictive modeling

pover — Fri, 12 Nov 2010 16:57:57 GMT

1) Very rarely. Consultant.
2) I've just done complex what-if's based on user-defined drivers and historical data to predict future results, but the what-if always has a user behind it making the changes according to what they see in their analysis. Also, sometimes, basic linear regression.
3) No

I agreed with John's and Lar's comments. This is an interesting topic and deserves more analysis and demos to add an extra punch to QlikView. A couple years ago I read a Gartner User Survey that contained a page that surprised me. This might be a copyright violation, but it's from the 2009 Gartner Report.

Predictive Modeling and Data Mining

The least used and lowest rated of any capability surveyed, with just 27% of the sample using predictive modeling and data mining, and a mean rating of 64.4 among those using them.

Statistical and predictive modeling

Thu, 09 Dec 2010 10:37:34 GMT

A bit late (but)

1) Not too often however if an Excel model needs to be updated regularly this is be moved into Qlikview. (In house)

2) Adding a seasonally adjusted forecast line to historical price data at various probability levels (P90-P50-P10).

3) Multiple regression using Excel Linest to calculate the regression parameters fed back into Qlikview.

QV really needs some multiple regression capability as it would be very powerful to dynamically update regression parameters based on the current selection.

Statistical and predictive modeling

Thu, 09 Dec 2010 21:30:22 GMT

Hi Gary,

As statistician, i'm very frustrated about the lack of statistics tools inside QV.

2) So I just use basic descriptive statistics (bar chart distribution, box-and-whisker diagram), sometime I add a trend curve in a time plot chart .

3) I use SAS very often for statistical purpose (mainly regression, factorial analysis or cluster analysis). The best thing I found out was to use the ODBC driver for SAS and to connect QV with SAS. It's very simple to set up and very useful.

Example : after a principal component analysis with SAS, I'm directly able to read the resulting file in QV then to display the principal component in a scatter plot.

Regards

Statistical and predictive modeling

Thu, 26 May 2011 22:43:09 GMT

What type of models are you looking for?

Qlikview aloud us to compute models in real time.

Most of statistic models made in spss and other statistic and econometrics softwares are executable in Qlikview.

The Problem is that we need to include all syntaxs in Qlikview and that is time consumming.

My experience is quite good because after um compute the model in Qlikview we can filter information and model recalculate the p values.

Best regards

Re: Statistical and predictive modeling

Anonymous — Fri, 27 May 2011 20:52:53 GMT

We have created some simple SPC Charts. Pretty simple to do. One needs to manually put the formuals in the expressions for the median, UCL and LCL limits

Statistical and predictive modeling

Tue, 06 Sep 2011 12:29:56 GMT

Theres a few things that you could developed using statistics. Off course its not a data mining tool, but there's a few analysis that your are to create such as Anova's, Outlier's, SPC Charts, Samples Means Comparisons, among others. Lets hope in a short-run future Qliktech developed data mining tools for qlikview...that will be real innovation

Statistical and predictive modeling

Mon, 19 Dec 2011 03:40:46 GMT

This is obviously late, but I would look for integrative ways to apply more robust statistical techniques to any analysis with some of the existing packages out there, if indeed this is a target market of yours. As previously mentioned, there is a way to take SAS output to QV. R/SPSS/Stata/Statistica/etc I can't really comment on (my experience is in R and SAS). I mention in another post of mine about the potential merits of providing visualizations for the purpose of further analyses, not merely reporting (Tableau might be an example here). However, there are companies who have taken R and merged with other open source products, such as JasperSofts products, to produce analytic throughput to an environment which can be delivered (even interactively) as a reporting product or an application (i.e. R script called in Java to produce, say a flexible set of visualizations).

I've been lurking the forum looking at the treatment of time series analysis, but it appears many users may be more interested in sticking to basic trending. It doesn't appear that QV has the appropriate tools to really go beyond this. Interestingly, opposed to financial risk analytic groups, most financial (accounting type) groups appear to gamble on very linear and simplified methods of making future predictions (of course the problem always comes when one cannot explain the degree of uncertainty with any results). This is quite similar to those who would use IBM Cognos, whose 'forecasting functionality' really doesn't belong in the space of robust analytics.

Alot of the direction in business analytics toward robust models is sadly slow to adopt - particularly depending on your target business. This is true for a number of reasons, which include - time to initial model development, statistical expertise of the analyst, the research questions that various businesses feel are sufficient to get information on, and really - the general business gestalt of the role the analyst should play in helping develop and refine the right questions before attemping to analyze them in the first place. These are truly fundamental issues that you kinda need to know up front before venturing into the data mining and statistical modeling space.

Just my $.02.

Phillip