Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
cheburashka
Creator III
Creator III

Incremental load of governance dashboard

Hello,

How does the incremental load of the governance dashboard work?

The fileScan sub seems to create a list of qvds and qvws. Does the incremental load check which qvws and qvds have been modified?

Or how does the incremental load work?

Where in the script is this done?

,Thanks for your help

twa

1 Solution

Accepted Solutions
Tyler_Waterfall
Employee
Employee

The whole incremental process involves checking for and loading the historical QVD for each "area" (like QVGD_SOE_2.0.1.qvd). With that loaded, there is now in memory a list of all the previously loaded files of that file type (QVDs or QVWs in this case).

The Governance Dashboard then creates a list of all the QVDs & QVW files in a folder and checks the filedate().

The file will then be read/analyzed in that reload if:

1- the QVD or QVW has a filedate() more recent than the start of the last reload of the Governance Dashboard

2- OR, the QVD/QVW does not already exist in the list of "read" QVWs/QVDs and the filedate() is within the cutoff date (the months of history set on the Configuration page)

Does this answer your question?

Tyler

View solution in original post

4 Replies
Tyler_Waterfall
Employee
Employee

The whole incremental process involves checking for and loading the historical QVD for each "area" (like QVGD_SOE_2.0.1.qvd). With that loaded, there is now in memory a list of all the previously loaded files of that file type (QVDs or QVWs in this case).

The Governance Dashboard then creates a list of all the QVDs & QVW files in a folder and checks the filedate().

The file will then be read/analyzed in that reload if:

1- the QVD or QVW has a filedate() more recent than the start of the last reload of the Governance Dashboard

2- OR, the QVD/QVW does not already exist in the list of "read" QVWs/QVDs and the filedate() is within the cutoff date (the months of history set on the Configuration page)

Does this answer your question?

Tyler

cheburashka
Creator III
Creator III
Author

Hello twa‌,

Example:

I have a file A with file date 01/01/2016 on my hardrive. On 02/01/2016 this file information will be read and put in the QVGD qvds. My vInputMonthsOfHistory (CutOff date) variable is set to 3. So I will keep 3 months of history.

Let's say on 1st of february I move file A to the archive. What I would like is that on that day my history timer starts counting for the information of file A. I want the data for that file to stay in the QVGD files for 3 months starting on the day I removed file A because maybe some other files for example dashboards that used the domain file A are still on the harddrive and in use. As I want information about that dashboard I also want to keep some information about the input files of that dashboard which is file A. But I don't want to keep the actual file A, as I already removed it.

I think the file date of a file is not that important. If the file B is for example one year old and still used regulary. The information for that file will not be available in my governance dashboard because the file will not be read because my cutof date is set to 6 months. As the file date is older then the cutoff date the file will not be read by the governance dashboard. It is clearly a file that I do want in my Gov Dashboard.

Conclusion of my reasoning:

I think their should be 2 History variables

1) NumberOfMonthsBeforeDrop used to calculate the drop date of a file.

Drop date = date of file last found on drive + NumberOfMonthsBeforeDrop

Every load it is checked if a file still exists on the harddrive and the "date of file last found on drive" is changed when the file is still there.

2) AgeLimitOfFiles which is used to decide which files which are actually on the harddrive are read.

File date of file < Today - AgeLimitOfFiles  --> Do not load file.

I think the AgeLimitOfFiles will not be used that often. So it should also be possible to turn it of.

Did I mis something? Is this already available in the dashboard?

,KR Koen

cheburashka
Creator III
Creator III
Author

I tried to solve this by setting

LET vInputMonthsOfHistory = if( today() = MonthStart(Today())   ,  3, 12) ;

If I'm correct at the beginning of each month, this will drop all historic data of files that are older then 3 months.

Information about files that are on drive and which are older then 3 months will not be read.

but the second day of the month they will and we will have the required information back.

,Koen

Tyler_Waterfall
Employee
Employee

Wow - this is sounding a bit more advanced than the current incremental logic supports.

You are welcome to try this out, but I do not see an easy way to implement the logic you are trying to accomplish.

Good luck!

Tyler