Qlik vs Datawarehouse

[Qlik-addict]

I am working for a company that believe all data modeling should be executed in the database environment and stopping all the qlik developers to do any scripting in Qlik.   

If we found the data model not working, we need to help the database team to make the data model right in the database environment.

If we refuse to do it, we are the one everyone will condemn for not being a good team player, and none of them will even think why the data model was NOT done correct at start? 

Many projects struggle because of chunky view these database 'expert' build into their views and making the reload slow or the app having too many table that basically come with duplicate records.

Honestly speaking I can do all the works in a few days if I am allowed to access the data straight from the source system but was again being condemned for breaking the rules and increase the risk where data being inconsistent with database.    Again, I can prove to them that the data inconsistency is actually is from the transformed data done in datawarehouse.  (anyway, I had tried it to access the raw data and get everything work out the way in should be but I just can't share the data as at this stage, I just felt like I'm the criminal for doing the right thing).   

Has anyone being in situation like this before and I want to hear your feedback about the right approach or what you have done or said that turn the situation around without putting other downs?  I know I should resign but I believe a lots of organisations are having the datawarehouse driving all reporting matter - wherever I go, I am  going to meet these ignorant database people to tell me and dictate the direction for reporting.

But I strongly believe Qlik Sense or Qlikview can do a much much better work than all the work they have invested and done in the datawarehouse - one thing for sure, we do not have to JOIN tables every single times.   Why people can't see this as the biggest threat in all data modeling?  

At the start, I thought I give them the reasonable of doubts and perhaps I overlook some really key benefits to have all data model build in datawarehouse like business rules all can be easily traced if everything are done in datawarehouse.   But i have been working with my company for quite sometimes and all I see is mess after mess and all i do is clearing their works all the times. 

And this bunch of people when there is issue, all the know is to add more people into the process, and the root cause of the issue totally ignore.  If anyone bring up, we will be the one being condemned for being negative and focus on the problem not the solution.   

Personally, I don't believe they can build all business rule in datawarehouse as most business rule involves field from facts with certain dimensions fields and that mean, there will be ALWAYS join involve between fact and dimension and they always be a risk to duplicate records.  Am I right?

Any suggestions are welcomed.   I want to hear real experience in project or method to resolve this conflict. 

Is there any organisation out there doing theirs reporting without going through the datawarehouse and using theirs transformed data ?

Is it worth the risk of messing up the data set to honor a vision to build all business rules in data warehouse ?  For those have the same vision, is it really your team is able to troubleshoot better and working more efficient?  

Shall data warehouse be the driver for all reporting requirements?

[mruehl · Accepted solution]

what you have done or said that turn the situation around without putting other downs?

Talk to each other. This isn't a philosophical discussion but one based on technical facts.
If the dwh-developers think they're right, let them prove.

Personally, I don't believe they can build all business rule in datawarehouse

Of course it is possible. Qlik isn't the only tool to build business rules.
Both has advantages and disadvantages. I always try to have one single source for the data.
No Excel-Bypasses, no inline-Tables except for design issues and so on. But mostly I define KPIs
within Qlik. On my opinion it is a matter of responsibility. As long as I can't control the business rules
I can't be responsible for reports.

Shall data warehouse be the driver for all reporting requirements?

Of course not, your (internal) customer should be the driver.

[tresesco · Accepted solution]

I can feel it and hear what you are saying. Let's take a deep breathe. 🙂

I agree with you in most of the points (expect for making it generic that data warehouse is not needed in qlik projects; because there are many cases where DWH is needed and comes very handy) and would vote for if there would be something like - 'do we really need to go via WH for every steps of transformation? ' - discussion forum. However, we have to understand that it is less about technology implementation right/wrong decision, but more about - 'who leads the BI in the organization' question. Because, in most organizations for business and management Qlik is just another BI tool, if that works fine (after a fair amount of time run, may be 3-5 years) then okay, else we have other renowned tools Tableau or PBI (PBI is penetrating like anything). But people like us who are a bit more biased to Qlik feel the pain when see it fails because of bad implementation. 

If you are a developer or senior developer in a big team/org - you don't have much to contribute on deployment architecture, that is already in place. If you are very good in convincing people and winning arguments then you may give a try to set up a meeting to discuss with respective team/people. But again it's very very difficult because you are trying to do what you are not hired to do and people might not only look for logical discussions in the meeting but also - come pre-prepared to just negate it  and unfortunately we all humans are so in mind. Therefore, without authority - pushing a change becomes next to impossible. And that is the reason I said, 'it's about who are leading the BI team' - the authority. If the authority is convinced, you wouldn't have to go through all the steps of ladder convincing, they can force it in a more suitable and acceptable way. But if your BI authority is not much qlik-technical, they would want you to follow the chain of convincing all - which brings a obvious failure; you become frustrated.  

Writing long, losing focus what I wanted to say. To wrap-up, It's difficult to bring a change in such established architecture especially when you are not at that level and that's not a given role to you; and BI leaders are not much  qlik-technical. You could discuss and make the team understand on differences between other BI tools and Qlik, especially the associative engine part. This might give them the understanding that everything is not required to be joined in warehouse table; they also might understand that for quick turn around time, they could skip few steps in dwh and do it in qvd or qlik app itself. It would take time.

Don't just resign, take your time.  

[Qlik-addict]

Thank you Mruehi for your short and straight the point reply, much appreciated.

Thank you Tresesco for your kind, honest and detailed experiences sharing.  

Both of you are spot on about the leadership, the accountability and the challenges of bringing change to an organisation. 

Yes.  change of organisation culture takes times and definitely not an individual effort.  Guess I will continue to be tormented by these bunch for a while.  so hard to be RIGHT in a group of WRONGs.

Cheers for the response!