Loading data best practices - Web service?

[jad_harb]

Hello,

I am checking whether it is a best practice to load data incrementally from a Web Service instead of accessing directly the source data base of a given information system. The incremental load would get the delta which represents a sample of 100 columns and 200 rows for a given table and it is executed every 10 minutes.

The idea of using web service would be to be decoupled from any data model change happening on the source application and make sure you always having a running QVD loader.

Your opinions on the pros and cons on this topic is highly appreciated and whether it is really recommended as best practice.

Jad

[Peter_Cammaert]

Loading from the source database is always faster, which is an important performance measure in QlikView based solutions.

In an RDBMS, you can define Views to isolate a changing DB schema from QlikView. You only need to update your view whenever something important changes in table layout, column values or table relations. The View interface is to be well-defined and delivers a consistent data stream across database changes.

[jad_harb]

Hello Peter,

Thank you for your input. Indeed the performance is an important measure.

Now for the view idea, it doesn't solve the dependency problem as you will need to be aware before hand if for any data model change. Am I correct?

Would you have any other idea or know of any best practice in the loading and incremental loading?

Thanks a lot for any hint on that matter,

Jad

[Peter_Cammaert · Accepted solution]

I think you misunderstood me.

The creation of a View in this case is intended to isolate all DB changes from a 3rd party (e.g. QlikView) that isn't really interested in what happens behind the scenes. It becomes the reponsibility of the DB designer/maintainer to keep the View in good working condition and functional according to the strict View definitions,. So that a 3rd party like QlikView is offered a stable and consistent interface across any DB changes or redesigns that may happen. Compare this to your Web service idea where you try to isolate the same changes from your client application.

The main advantage of ssuch a View is that performance is optimal because of being implemented inside the RDBMS. The main disadvantage is that you'll have to talk to the DB maintainer if that's not you.

[jad_harb]

Hello Peter,

I understand you better now!

This makes total sense.

Thank you for your response,

Jad