Skip to main content
Announcements
Qlik Connect 2024! Seize endless possibilities! LEARN MORE
cancel
Showing results for 
Search instead for 
Did you mean: 
AlfonsS
Contributor
Contributor

Best Practices for multiple databases with similar structure

Hi,

the situation is as follows:  we have a data model with various tables (taken from Microsoft NAV).
We also have several clients for every country. Every branch has more or less the same data model, save a few tables here and there.
My question now would be, what are considered best practices regarding reloading / storing qvds for this?

We do have a legacy report we took over, with a somewhat convoluted "QVD Loader" file, that stores qvds in folders for each of those branches.
It also does some sort of pre-processing, so the qvds aren't exact images of the database tables.
The reports themselves use variables to differentiate between concatonating and creating a table for the first time, which doesn't exactly help with readability.


We are in the process of rebuilding that structure,
however, and now I'm wondering what would be the best way to handle this. Do I just store the tables 1:1 in .qvd files (so I can e.g. use incremental loading)
and do all the logic in the actual reports? Do I keep that logic in the backend so the reports themselves don't have to do as much heavy lifting?
Is it better practice to have one "QVD loader" that loops through all of the branches, or a seperate one for each branch?

I'd also appreciate general thoughts and ideas of how to handle this.

Thanks!

1 Reply
marcus_sommer

Like always it depends to various factors which measures are mandatory respectively suitable in regard of the efforts to develop and maintain an environment, the amount of data and user, the available hardware-resources and the requirements of the business.

The more division of work is applied the simpler are the single parts and the more could they be distributed - of course adding with each layer and each part a bit more overhead. To find the right balance here could be difficult.

Your use-case seems not to be a small one else implementing an entire environment for a larger reporting. Therefore I suggest to use at least a 3-tier data-architecture - means in Qlik usually: Generators --> Datamodels --> Reports. Depending on the amount and kind of the needed transformations you may apply some more layers whereby it's not mandatory necessary to put each cascade into an own layer.

Personally I use a mix of 4 layers with a split of the generators in a more extract- and a more transformation layer - whereby the extract-layer isn't 1:1 else adds some keys, formatting, filter, mapping, cleaning and so on while the transformation-layer makes the more heavier transformations.
But it's not a very strict differentiating else mostly pragmatic depending on the specific requirement. Nearly each different data-structure gets its own extracting/transforming because it keeps the needed logic small, simple and readable whereby everything what is a bit heavier get an incremental logic (not only the extractions else the transformations, too).
Of course you could combine multiple logics and making them also completely dynamic by pre-loading the data-structure and/or various meta-data to build loadings on the fly but in most cases you won't save efforts and load-times else rather the opposite happens.

Beside this try to put all logic within the script and developing the datamodels in the direction of a star-scheme with a single fact-table and n dimension-tables.

- Marcus