Real-Time Data Processing

[#1]

(I'm posting this again as the original took so long to be accepted that no one read it!)  🙂

Hiya Guys,

I am new to QlikView and have been reading (briefly) on the ability to show real-time data on a 'live' Dashboard via an add-on/activating a different type of License(?).

As I am learning QV as it is - I don't really have the time to delve too deeply into the workings of the real-time aspect, so my question is:

Am I able to create my .qvd and .qvw files in the standard way (for scheduled data refresh, say, midnight every day) and then "bolt-on" the real-time processing aspect to update the data instantly at a later date?... or is it an entirely different methodology which I need to understand before designing my datasets and Dashboards?

Any help you can provide will be greatly appreciated.

Cheers,

Steve.

[swuehl]

Steve,

as far as I know, QT will stop marketing a separate real-time add-on and are going to implement this as standard feature into the servers. (I think, since QV11 SR1, this has already happened:

http://community.qlik.com/docs/DOC-2581#How_does_QlikView_11_address_realtime_data_access

)

I haven't used this feature personally, so I can't really comment on this based on experience (and should probably just stop here).

What I learned from the forum, this feature allows you to modify the intenal data model after your script load (well, more precisely, I believe it only allows you to modify data values in existing table fields, like deleting, adding, modifying  records). And yes, you should be able to create a qvw file as you are used to.

As far as I know, this update is not really performant, because QV will need to clean up its internal data storage after each change. So it is only recommend for very small changes to your data and not at high speed.

Search the forum for real time server and dynamic data update. There are also Dynamic Data actions, so you could play around a bit:

http://community.qlik.com/message/204046

Regards,

Stefan

[#3]

(I've tried to reply to this 3 times now - and it won't let me... so I'm writing this AGAIN on the off-chance it gets published!)

Hiya swuehl,

Thank you for your reply - I've had a look through the links and they're quite interesting!

However, the company I work for is using QlikView 10 - so I am going to have to research the workings of real-time processing using the earlier version.  I have struggled to find any information so far - so if you have any ideas please let me know!  🙂

The dashboard I am looking to develop is for a small IT Helpdesk ‘ticketing’ system (incoming calls, SLA timers, closed calls, etc.) and it's MS Access database is only 12MB (even before I create a data warehouse based solely on the tables required!) - so I can't see it being very data-intensive, nor particularly high-speed.

Again, if anyone has any more information or ideas then please let me know!

Cheers,

Steve.

[#4 · Accepted solution]

Just as an update I'll post my solution so far:

We have now upgraded to QV11 - and within QEMC there is a "Continuous Reload" option under "Documents".  Despite the entire Dashboard (and associated .qvw files) being less than 900KB - this caused us some issues during data refresh (even though the entire reload would typically take less than 30-seconds - the MS Access database it was accessing would tie itself in knots).

I have now set the reload to occur once every minute - and this seems to be working fine (with Document Properties: "Server performs refresh automatically without client action", "Enable Push from Server", and "Allow Dynamic Data Update" all selected).  As long as the Dashboard is opened using the IE Plugin via the Access Point the refresh is seamless and all appears to be working well.

Development time is short at the moment - so I am just reloading all data from scratch each time... although this is possibly something which could be improved with an 'incremental load' in the future.

If anyone has any other thoughts/comments they are most welcome - and once I am confident that this is a stable solution I will mark this as "Correct" so it will hopefully help others in future.

Cheers,

Steve.

[#5]

Hi Steve,

You obtained some results about your research of Real-Time data processing?

[#6]

Hiya vieiras8,

I have found that the best solution is the one above.

However I have made a few enhancements:

1)  Migrating the database from MS Access to SQL Server stops the ODBC connection locking - and so using "Continuous Reload" is now not a problem;

2)  Moving the dashboard to the same physical Server reduced load-time from 30-seconds down to 4-seconds!  So the result now is as close to real-time as I'm going to get;

The only issue I have with "Continuous Reload" is that there is no scheduling available - so, even though we only need the Dahboard between the hours of 08:00-18:00, the data is reloaded and refreshed all throughout the night.  I have also played with setting-up a ".bat" file to reload the data via Windows Task Scheduler (which works well) however it takes the reload slightly longer to complete - and the lowest frequency possible is only once every minute.

I hope this has helped!

Cheers,

Steve.

[#7]

Useful answer for my research! Superb! Thank you! I will write you some questions later!

cheers,
Yago

[Anonymous]

Hi Steve

I reckon OLEDB based database connecitviy is much faster than ODBC driver esp for this type of real time data processing.

Regards

MultiView

[#9]

Hiya Multi View,

Thanks for the info... I tired both methods, and with the dataset being so small I couldn't record any performance difference - and unfortunately it also didn't solve the 'connection locking' issue with MS Access either.

It is something which may be worth changing in the future - I assume it's just the Connection String which needs rewriting?

Cheers,

Steve.

[#10]

@ Steven Bain

   I enabled the Allow Dynamic Data update, For POC purporse i added 4 data at the excel, then ever 10 minutes i add few more data, I could not able to see any out put in the Qlikview report. Any thing more i need to do..

Kind note : I never Schedule the report.