4 Replies Latest reply: Aug 1, 2016 10:38 AM by Tyler Waterfall RSS

    Incremental load of governance dashboard

    Koen Bal



      How does the incremental load of the governance dashboard work?

      The fileScan sub seems to create a list of qvds and qvws. Does the incremental load check which qvws and qvds have been modified?

      Or how does the incremental load work?

      Where in the script is this done?


      ,Thanks for your help


        • Re: Incremental load of governance dashboard
          Tyler Waterfall

          The whole incremental process involves checking for and loading the historical QVD for each "area" (like QVGD_SOE_2.0.1.qvd). With that loaded, there is now in memory a list of all the previously loaded files of that file type (QVDs or QVWs in this case).


          The Governance Dashboard then creates a list of all the QVDs & QVW files in a folder and checks the filedate().

          The file will then be read/analyzed in that reload if:

          1- the QVD or QVW has a filedate() more recent than the start of the last reload of the Governance Dashboard

          2- OR, the QVD/QVW does not already exist in the list of "read" QVWs/QVDs and the filedate() is within the cutoff date (the months of history set on the Configuration page)


          Does this answer your question?


            • Re: Incremental load of governance dashboard
              Koen Bal

              Hello twa,



              I have a file A with file date 01/01/2016 on my hardrive. On 02/01/2016 this file information will be read and put in the QVGD qvds. My vInputMonthsOfHistory (CutOff date) variable is set to 3. So I will keep 3 months of history.


              Let's say on 1st of february I move file A to the archive. What I would like is that on that day my history timer starts counting for the information of file A. I want the data for that file to stay in the QVGD files for 3 months starting on the day I removed file A because maybe some other files for example dashboards that used the domain file A are still on the harddrive and in use. As I want information about that dashboard I also want to keep some information about the input files of that dashboard which is file A. But I don't want to keep the actual file A, as I already removed it.


              I think the file date of a file is not that important. If the file B is for example one year old and still used regulary. The information for that file will not be available in my governance dashboard because the file will not be read because my cutof date is set to 6 months. As the file date is older then the cutoff date the file will not be read by the governance dashboard. It is clearly a file that I do want in my Gov Dashboard.


              Conclusion of my reasoning:

              I think their should be 2 History variables

              1) NumberOfMonthsBeforeDrop used to calculate the drop date of a file.

              Drop date = date of file last found on drive + NumberOfMonthsBeforeDrop

              Every load it is checked if a file still exists on the harddrive and the "date of file last found on drive" is changed when the file is still there.


              2) AgeLimitOfFiles which is used to decide which files which are actually on the harddrive are read.

              File date of file < Today - AgeLimitOfFiles  --> Do not load file.

              I think the AgeLimitOfFiles will not be used that often. So it should also be possible to turn it of.


              Did I mis something? Is this already available in the dashboard?


              ,KR Koen

                • Re: Incremental load of governance dashboard
                  Koen Bal

                  I tried to solve this by setting

                  LET vInputMonthsOfHistory = if( today() = MonthStart(Today())   ,  3, 12) ;


                  If I'm correct at the beginning of each month, this will drop all historic data of files that are older then 3 months.

                  Information about files that are on drive and which are older then 3 months will not be read.

                  but the second day of the month they will and we will have the required information back.