I was just asked the other day if I had any quantitative way of measuring the value of some of the applications that we have built. What a complicated thing to do in our line of work.
Let's say that we build an application that enables the company to effectively do staffing on demand. If that effort saves the company 3,390 man hours per week then we obviously have an easily measured ROI right? It costs $10 for my salary, it costs $20 for Qlikview licenses and servers and we save 18 million per year in salaries. Love it.
But what if that same application was intended more to improve patient care and reduce bottlenecks and thus we demonstrate that we need 3,390 additional man hours per year at these exact hours of the day and days per week in order to speed up the clinical work flow to discharge patients sooner and increase patient satisfaction levels. Bummers nobody is standing in line to put bonus money into our pockets now.
Obviously we had a huge impact but even something like dollars that any other industry can quantify isn't really a sure fire indicator of our "effectiveness."
My next blog post, selfless promotion. is going to be on "Visualizing User Adoption." One of the things I'm going to discuss is this very topic. What does "adoption" look like? Is it the number of people that use the application? Wow I got 193 people using the application. Sounds good right. What if they all use it for 1 minute per day and only have 1 click each? Is that truly an "analytical application." Or is it a pretty Qlikview rendition of a canned report that probably could have been emailed to them straight out of the databse.
Does adoption involve the amount of minutes that users spend in the application? That's more straight forward right? Except do we then need to start looking at averages or medians instead of just the total? If we do that do we need to exclude outliers? Or is an outlier really a good thing in terms of using an analytical application. Meaning why should I exclude that analytic zombie who spends 12 hours straight working in an application?
For sales a trend is always a great way to measure effectiveness. But is it a good way to measure an analytical application. Upper management will love you when you show them a positive trend "look boss this application had 100 minutes of usage in January, 200 in February, 400 in March etc." But what if your application had an upward trend for a short period and now is trending downward? is that a sign of poor performer or is that the sign of a great performer. You enabled the company to recognize and take action on a poorly performing process and now it is no longer a real issue?
But how do we know that we should have more of users, or minutes? The applications we build are constructed with completely different purposes and have far different groups consuming them. You may build the most important application in the companies history but the target group is only 5 people. While I may build an application to show people how much vacation time they've accumulated. A statistic they could connect 18 ways to Sunday to see but they use my application.
hen I'm looking at how I would measure these things, I become an Analyst myself. Only the data is my own. So how do I avoid choosing the measurements metrics that make me look best. I see a pattern for my favorite 3 applications, the ones I'm most proud of, the ones I worked on the most and so I choose the measurements that make those applications look best. Human nature. I'm guilty.
I just read a very interesting study someone had shared about forming 30 analytic teams and giving them a problem to solve. They were asked to prove if a certain situation proved/disproved a theory that involved race bias in sports in some way. My problem with the study was that what I believed it really proved was that the people working tried to use the data to prove their own bias in one way or another. What I think they should have done was blinded the data in a way that the analysts had no idea what they were proving or disproving. And simply used the data. Now I can't even trust myself to look at data in a Session.Log fairly.
If this were sales we would build a budget ... we expect that Dan's application should be used by 5 people and that they will use it 3 times per week for 15 minutes each time. But that's impossible for us. If you knew it would take 5 people spending 45 minutes each week to solve the problem, then you'd build a better application that just solved the problem. We can't even guess how long it should take people to do analysis. We can't guess how many people may catch on and become really good at analyzing the data. We can't predict how many problems we will surface through our tools that weren't even being asked about when we got started.
Crud ... now I just gave away the points for my blog post and I'll have at least 1 fewer readers. Now how do I measure the worthiness of my post?
On a serious note Dan ... I would love to speak with you about your thoughts on this. As I'm seriously writing about this over the next few days. I would love to get your input and thoughts. I love that this community is actually beginning to move in a direction where we are all seeking answers to some really tough questions. My cell is 678-943-0064 if you do have some time to chat this through.
