Skip to main content
Announcements
Introducing Qlik Answers: A plug-and-play, Generative AI powered RAG solution. READ ALL ABOUT IT!
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Database or QlikView: Suggestions and Feedback

Hello every one,

I know this can not be generalize but over all experience on working with large amount of data on QlikeView. I need suggestions and feedback for cleaner and faster (Optimized) way to do followings.

1. Which one is better (clear and faster) to do data formatting and manipulation in

database like sql server / oracle and then load data in Qlikview or load data in to

Qlikview and then do the formatting and manipulation in QlikView. What is general

take?

2. Extract data from multiple sources then store in the database and extract to QlikView or extract data directly to QlikView and then do the processing. What is the ffedback.

3. Is it a good idea to store raw data in qvd and then do formatting and manipulation of

data in QVW or store manipulated data in QVD and then load in QVD.

I hope I am clear in explaining my questions but want to take general idea from all experience fellows.

Thank you in advance.

3 Replies
Not applicable
Author

I can't give you a consensus answer on these questions, but I can describe how my organization is using QlikView.

We have a mature data warehouse. Everything is being processed into the data warehouse before being used in QlikView. We then create QVDs with our various tables and then load them into our applications. On the application end, I am trying to the least amount of processing possible as I like to be able to do Optimized loads. Therefore, I am doing most of the data manipulation in the QVD creation phase.

If we didn't have a data warehouse, I'm not sure how we would handle it. QlikView should be able to pull in your various sources without the need for a data warehouse. I think without the data warehouse, I would let QlikView do the processing without first bringing everything together into a database.

I guess in summary, having all of your data already housed in a database is helpful, but unnecessary as QlikView can handle all of that for you, if needed.

johnw
Champion III
Champion III

Here are my opinions, but they're just opinions. There are lots of different approaches that can work.

1) Your databases should serve the needs of your business systems, not your reporting system. Your reporting system (QlikView) should then do the data manipulation and formatting necessary for its reporting.

2) Extract to QlikView directly. Again, the databases serve the business system, not the reporting system, and QlikView is perfectly capable of doing the data manipulations required to translate for its own needs.

3) I store fairly but not completely raw data in the QVDs. Let's take a hypothetical example. In our business system, we store both customers and suppliers. We might have three different tables. One is a "business entity" table that stores information that all of our business entities need, with a cryptic field like BEID as our ID. Then we might have a separate table for customer-specific information, and a third table for supplier-specific information, with IDs of CUID and SUID. A lot of the fields on these tables might be codes to be looked up on yet another database. Like you might have a CRST (credit status) field with a value of "03" that, when looked up on the other table, really means "require payment up front".

I would NOT try to replicate this structure when building QVDs. In a sense, I would shift to a more object-oriented approach, in this case, having a customer object and a supplier object. So I would create a Customer.qvd and a Supplier.qvd. Rather than the cryptic field names, I would use "Customer" as the customer ID, and "Supplier" as the supplier ID, as well as naming all of the other important fields something more human-readable. I would only include fields that people wanted to see on reports, not all fields in the database. If someone wanted to see the information, I would go ahead and add field "require payment up front?" with Y/N values to the Customer QVD, rather than storing a separate table to look up this one piece of data. Basically, QlikView's compression makes this sort of denormalization a non-issue. I'd also put date fields in QlikView's date format, and that kind of thing.

I wouldn't personally go any farther than that, and we don't in our shop. I'm a fan of having the QVDs be very basic objects with little manipulation. I feel that gives them the greatest chance of being useful to the greatest number of reporting applications. But if you find yourself doing the same manipulations over and over in the applications, by all means do them when creating the QVD rather than giving yourself a maintenance nightmare of having very similar script in a number of different applications.

In regards to optimized loads, I've recently started doing all optimized loads with my most restrictive possible exists(), and then doing additional manipulation in memory. For instance, then inner joining to an inline load of allowed values for some field. That's worked well so far, but I can't say for sure that it will continue working well. Most of my existing applications use unoptimized loads and are quite slow to load. Up to now, my concern has been much more for execution speed while someone is waiting for a chart than load speed, so I'm not really a load speed expert.

Not applicable
Author

Thank you for your reply.