gerryhorgan wrote:your answer surprises me, since you are obviously experienced and very vocal on this site. I don't think you gave a straight answer, rather a pedantic one that misrepresented the product.
I gave a straight answer as best I could. But as I said to the original poster, "I do think I'm misunderstanding your questions." And perhaps I'm misunderstanding your points about ETL as well. But my answer wasn't a sales pitch - I couldn't care less if QlikTech or some other company gets your money, QlikView isn't the answer to all problems, and there are plenty of things about QlikView that I'm not happy with. I'm not out to mislead anyone about what QlikView can and can't do or what it should or shouldn't be used for.
gerryhorgan wrote:I believe that QlikView have chosen to gloss over this with deliberate half truths. I'm specifically referring to their web page "QlikView and your data" with the section "Beyond traditional ETL - Add value not redundancy". Paraphrasing, they state that traditional BI requires ETL followed by cube building. They then state that due to Qlikview's in-memory architecture, cubes are not required and infer that neither is ETL since "data simply needs to be moved in to memory". I agree about the cubes. I disagree about most of the ETL data preparation leading up to the time of cube construction. I see this as a deliberate half truth from QlikView. They go on to say that "qlikView can aggregate and calculate in real-time". So what? What I interpret this as is, QlikView can aggregate and calculate good data intermingled with transactional garbage in real time.
Let me start with this bit, since I mostly agree with you here. You can't skip the ETL step for anything complicated. Merely aggregating transactional garbage is of little use. For complicated data, you either need a data warehouse (which required ETL to build) or you need to do ETL to get the data in a usable format for QlikView. You're very unlikely to be able to take complicated data from a modern business, load it in raw from databases, and get something beautiful in QlikView. If QlikTech implies that's how it works and that it's easy and wonderful, then I disagree with them. And if you think that's what I was implying, that was certainly not my intent. You need an ETL step.
gerryhorgan wrote:My limited personal experience with the product coupled with answers from both the sales and technical team during our proof of concept and our trainer, point to a different conclusion. QlikView is only suitable for light ETL or to put it another way QlikView is a bad ETL tool...
My conclusion which was agreed with grudgingly by the senior sales person on site is that the most efficient way to develop an overall QlikView solution is to perform a separate ETL stage before the data is fed to QlikView. QlikView is not an efficient way to do the many ETL tasks (even though some of them may be achieveable in an inefficient manner). It is not reasonable to expect users to access raw transactional data, use QlikView and produce totals that agree with existing reports (which may incorporate multiple complex business rules).
Well, I would disagree with your limited personal experience and with your senior sales person then. Now, perhaps you're right about your own ETL needs, but at our company, a separate ETL stage outside of QlikView is usually not the best or most efficient way to handle things in my opinion.
Initially, we did use separate ETL if I understand what you're saying. We had a handful of batch jobs, SAS jobs, whatever, that would actually do the database reading, transform the data into a reporting format, and then dump it out as a flat file. Several of these are still in existence. QlikView isn't serving as much of an ETL in those cases, just an L.
But as I gained experience, in some cases I realized that I could do that work in the QlikView script, and typically with MUCH less effort, many fewer lines of code. The problem was simply one of familiarity, in the sense that if you normally program in Java, you'll find programming in LISP to be very unfamiliar, very awkward. That doesn't mean that the system you're unfamiliar with is worse for the job in any objective sense. At this point, I do most new ETL inside of QlikView itself.
I do agree that in some cases, such as when all your raw data contains is some basic business facts, and rules about how to transform those facts into a report, QlikView is probably not the place to apply those rules. I'd probably encourage people in this case to apply those business rules to the business facts and actually store the results in separate tables. Perhaps think of it like a little data warehouse for that specific data, reporting data built from the transactional data and business rules. Then the reports and QlikView would load from there, and both would match because they're showing the same data. But in this case, the ETL is very much happening outside of QlikView, which is perhaps one of your main points. If so, yes, agreed. In some cases it is much better to do the ETL outside of QlikView. Complicated business logic isn't something you generally want to duplicate across multiple environments. You want to do it once and only once.
And even when in my opinion the ETL should be in QlikView, I'll agree that people are not going to be able to purchase QlikView and immediately start writing complicated ETL inside of it. It's going to take experience, possibly years of experience to master a new environment. In the mean time, writing complicated ETL in whatever system you're used to writing in may be the best solution. And you certainly CAN continue using QlikView in that way, as only a load step, with the actual extract and transformation occurring elsewhere. It's not my preference, but you can.
And I know of at least one QlikView expert (and I'm sure there are others) who would probably strongly disagree with me, and would probably state emphatically that the ETL should be done outside of QlikView, and that QlikView should be used as a reporting system only, not as an ETL system.
So returning now to me being misleading and pedantic, and perhaps being pedantic about it, nobody asked, "Can I do the most complicated possible ETL in QlikView, and do it easily, and is it the right place for a beginner to do it?" They just asked, "Can QlikView be considered an ETL." Simple ETL is simple. Complicated ETL is complicated. But it'll do both, I do both with it, and I generally prefer to do my ETL inside of QlikView instead of outside. I consider it an ETL product.
