Performance Question

[#1]

If we have a large amount of data (millions of rows), is there a way to limit the data that is sent to the client through a paging or other method?

Concerns around the amount of data that would be sent to the desktop - especially from a WAN.

[Oleg_Troyansky]

Well, you have to apply a lot of caution when dealing with large amounts of data. Here are some of the guidelines:

1. Place your Server that gathers the data from the Database, in the same geographic location as your data source. For example, if your database is in Europe, don't try pulling the data using a computer in America. Even if you data is in the other building 10 miles away - find a possibility to place your QlikView machine right next to it. Once the data is pulled and stored in QVD, you are better off copying (FTP if needed) the QVD files between locations than entertaining the load over the WAN.

Regarding paging - it depends on your database and ODBC driver. Some databases and their ODBC providers (AS/400 DB/2 for example) do a very good job managing the sizes of the individual packages, while some other databases/ODBC providers (Postgress as an extreme example) don't. SAP, for example, is using a "dumb" (in my mind) practice of abandoning your query is you are trying to pull more than a 1 million rows from a BW cube - that's just crazy!

The only way you can manually "pace" your downloads (if you can't configure it in the ODBC data source) is by selecting slices of data using SQL WHERE clause and building your data this way. For example, if you have million of rows in your Sales table, you can pull the data year by year, or month by month and limit your data size this way...

hope it all makes sense...

Oleg

[rwunderlich]

If the client is opening the document from Qlikview Server, the dataset is not sent to the client machine. Limited data is sent to the client to create the visual display. I don't know all the details, perhaps someone else can expand on this.

-Rob

[Oleg_Troyansky]

Rob,

you are right - perhaps, I misread the question... I addressed the problem of loading millions of rows from the database to the "desktop" of the Developer. If the question is related to the run-time and using Client and Server - no problem at all! As you mentioned, very small portion of the data is being send to the client from the server.

Oleg

[#5]

Hi,

There are two way of doing this,

a. As suggested above by Rob or

b. If can able to implement the security so that specific user can see specific set of data, that will help. Not sure wether this is applicable in your case.

Regards,

Amit

[#6]

Oleg,

My concern is around the data being sent from the dashboard to the client. If I bring down a high number of rows to the desktop (e.g. 1/2 million), I assume this would be problematic. Is there a way that the Dashboard can page the data that it sends to the desktop?

Thanks

Paul

[#7]

Paul,

Just to clarify, you'd have something like a straight/pivot table with 500K rows? Does the end user really want to scroll through that? Wouldn't you select down to a more reasonable number of interest to the user?

Tom

[Oleg_Troyansky]

Paul,

what Rob, Tom and myself are trying to say is - if you have several millions of rows in your application, it's OK, QlikView won't send them all to the desktop. To Tom's point, QlikView will send to the desktop the aggregated results required for rendering your charts and tables. So, if you have a very detailed table on your Dashboard (like a Picot Table or a Straight Table), that might cause some traffic problems. On the other side - you probably don't need a very detailed table on your dashboard - after all, we are talking about analytics and not detailed reporting.

does it make sense ?

Oleg