Single QVD Loader or Multiple loaders

[markodonovan]

Hi there,

I have come across a question that I would be very interested in hearing other qlikviewers opinions on.

The questions is to do we creating qlikview loaders.

If you were creating a loader that was going to be reading 100+ tables from 1 system would you put the code that would generate the initial source and transformed qvds into 1 qvw file ?

Each table would have a separate tab in the loadscript.

Have you any loadscripts with more than 100 tabs?

My gut instinct is to try and separate the scripts logically in some way but different people have different ideas.

What is your opinion ?

Thanks

Mark

[marcus_sommer · Accepted solution]

I would split these loads into several qvw's then the handling and the overwiew with so many loads and script-tabs is possible but surely it isn't really good. Further qv will be significant slower while editing, scrolling and so on in large and complex load-statements.

Also with several qvw's you could parallelize these loads and updating as well.

- Marcus

[giakoum]

I guess that when reading 100+ tables, they all or most of them contain fact that you concatenate into one large fact table. So I guess that many of these tables have the same fields. If yes, then I would not create 100 tabs, I would create one tab with code that iterates over the tables and loads them. Something like a for loop :

For each table in tables

     [$(table)]:

     load fields from $(table);

     store [$(table)] into $(table).qvd (qvd):

next

[markodonovan]

Hi Ioannis,

The script in each tab is currently doing an incremental qvd load.

The tables are being kept separate and all the fields are different in each table.

I nice idea though.

Thanks

Mark

[giakoum]

Is it that the names of the fields are different but they contain the same data?

For example customerID in Table1 is called FieldA and in table2 it is called CustID?

[markodonovan]

Hi Ioannis,

Thanks for the reply.

All the tables have different data.  I wish it was that simple.

My main worry was a code bloat scenario where there was a desire to put the code for all the source qvd loads and transformed qvds in 1 qvw document. 

The system we are getting the data from is quite large and having 100 tabs would easily be possible.

It is interested to hear if anyone else encounters such problems.

I appreciate the fast reply to this question.

Thanks

Mark

[giakoum]

I see.

Then I would suggest you use an Excel file to keep tables and their respective fields and use a for loop to iterate through them, load and store the data in qvds:

Excel sample:

Table, Field

Table1, Field1

Table1, Field2

Table1, Field3

Table2, FieldA

Table2, FieldB

etc.

So instead of maintaining 100 tabs, you maintain one large excel sheet and one tab in QV

[#8]

Can you elaborate on this approach? I'm in the process of deciding this myself, and like this idea, but am not sure exactly how to implement it. I'm worried about adding multiple tabs vs also tracking multiple qvd generator qvws. My one qvd generator right now is a bit slow with a little over 6M rows...