18 Replies Latest reply: Oct 18, 2017 12:23 PM by Rob Wunderlich RSS

    Governance_Dashboard - identify fields again ...

    Friedrich Hofmann

       

      Hi,

       

      I am still working on extracting parts of the available data from the „standard GovernanceDashboard“ running once a day and building our own customized version on that basis.

       

      I have just figured out where I can get a lot of data about all the tasks scheduled in the QEMC. That is very good because we really do have a lot of tasks there. There is one info that I cannot get from the GovernanceDashboard, the order in which Jobs in the QEMC must run - simply because we have not set any Tasks to run in sequence, but they all have separate time-dependent Triggers. I am working on putting together that Piece of info in an Excel file which I can then link to what I get from the GovernanceDashboard.

      Only one piece of information is missing – the duration of a specific task. That is in the log, visible in the QEMC, so it must be somewhere. But we do have that in a "Monitoring.txt" generated by every app - which simply contains one timestamp at the start of the script and one at the end - which I could also use to decide whether or not a specific app was executed on a given day.

       

      Maybe someone can explain to me what exactly is in the field "StartAt" in the table "PubDocTask"? There seem to be several pieces of Information in there. I will also try to find an answer or to Piece it together looking at the corresp. Tasks in our QEMC.

       

       

      Thanks a lot!

       

      Best regards,

       

       

       

      DataNibbler 

       

        • Re: Governance_Dashboard - identify fields again ...
          Friedrich Hofmann

          Well, the first part is a date, the date that is set as start_date for a schedule, that's clear.

          The "T" probably means that the Trigger is time-dependent and the following time is the set execution_time, also clear. Only the last bit, that "+02:00" (or similar) is not yet clear to me ...

          • Re: Governance_Dashboard - identify fields again ...
            Rob Wunderlich

            You can get a lot of task information (but not duration, sorry) from

             

            Qlikview Cookbook: QVPR Analysis http://qlikviewcookbook.com/recipes/download-info/qvpr-analysis/

             

            Perhaps you can find some info to extract there or something to help your analysis.

             

            -Rob

            http://masterssummit.com

            http://qlikviewcookbook.com

              • Re: Governance_Dashboard - identify fields again ...
                Friedrich Hofmann

                Thanks a lot Rob!

                 

                I will have a look.

                 

                Meanwhile, I am analyzing the "Filelist" generated by the GovernanceDashboard which Shows me all the qvw- and qvd-files generated.

                Am I right in assuming that the fields "LoadDate" and "LoadDateNum" are valid for qvw, but not for qvd files whereas the fields "FileDate" and "FileTime" are valid for qvd files? It seems so ...

                  • Re: Governance_Dashboard - identify fields again ...
                    Tyler Waterfall

                    As I recall, LoadDate and LoadDateNum relate to the actual reload of the Governance Dashboard. So if you reload the dashboard each day for 21 days, you would see 21 distinct values of LoadDate. These relate to the file metadata (QVD only I think) over time - allowing you to see QVD filesize over each of the 21 days you reloaded the Governance Dashboard.

                     

                    FileDate and FileTime are the actual file time (last saved/modified) for the QVWs and QVDs.

                      • Re: Governance_Dashboard - identify fields again ...
                        Friedrich Hofmann

                        Thanks a lot Tyler!

                         

                        I don't really think so, but I will try that out. Most of our apps actually have to run every day, most of them run in the early morning. I will have a Close look at the times. I can tell from the QEMC exactly when the GD is reloaded every day. In the "FileList" (a table), I have one "LoadDate" for every record - those would all be the same if that was always just the ReloadTime of the GovernanceDashboard itself, no? I will see.

                        I'll take one step at a time. Right now I'm working on gettting the ScriptDuration of every app running on our Server.

                        Best regards,

                         

                        DataNibbler

                          • Re: Governance_Dashboard - identify fields again ...
                            Friedrich Hofmann

                            Hmmm ... I can see the exact start and end time of a Task in the QEMC. Why does the QVGD not capture this - or does it? I could not find this info anywhere up to now. I did find a table in the QVGD which seems to list all the Jobs we have running in the QEMC, I have a start time of every Task, though only the scheduled one, but I do not have the Duration ...

                            True, I could deduce the Duration once I have associated a qvw with all the qvd_files it generates, if any, and their filetime - the last filetime is logically the very last Thing a script does, so if I have that and a start_time, I can deduce the Duration - only not all qvw_files generate any qvd_files.

                            My current Approach is having a few lines of code to capture that in the script of every app itself - it would just be easier and somewhat more elegant if I could get that directly from the Server.

                            I will have another look at the QVGD and what I can find online and I will have a look at what Rob suggested - though he says that Duration is not in there - but maybe someone has already had the same struggle and figured out how to do it?

                            Thanks a lot!

                            Best regards,

                             

                            DataNibbler

                              • Re: Governance_Dashboard - identify fields again ...
                                Rob Wunderlich

                                I believe QMC gets it's execution duration and history from the DIstributonService folders:

                                 

                                TaskExecutionHistory

                                TaskResults

                                 

                                You should be able to get the data you want from there.

                                 

                                -Rob

                                http://masterssummit.com

                                http://qlikviewcookbook.com

                                  • Re: Governance_Dashboard - identify fields again ...
                                    Friedrich Hofmann

                                    Hi Rob,

                                     

                                    thanks a lot!

                                    if that is so, there's no way I can currently get those Infos - the subdirectories of the "DistributionService" Directory are all empty. There are no logs there since we have not turned on the logging functionality in QlikView_Server ;-) The other subdirectories are empty, too, I guess that's for the same reason.

                                     

                                    The Thing is a bit tricky because we have been having Trouble with our Server quite frequently in the last few weeks and we are yet unable to exactly identify what is the reason, I guess those few lines of code needed in each app to write that small txt file - implemented as an INCLUDE to Keep it all consistent - is way less resource-heavy than another function to be carried out by the Server,  but well, we are at  loss yet as to what are the reasons - which is not being helped by the fact that the Server itself is located in FFM and administrated by another Company as usual ...

                                    Best regards,

                                     

                                    DataNibbler

                                      • Re: Governance_Dashboard - identify fields again ...
                                        Rob Wunderlich

                                        If the subdirectories of DistributionService are empty, then I believe you are looking at the wrong directory.  Those folders I referenced are not optional, they are not controlled by logging settings.

                                         

                                        The location of all DistributionService files is set in the QMC in the System, Distribution Service, General, "Application Data Folder" setting.  The default is "C:\ProgramData\QlikTech\DistributionService" but it can be changed.

                                         

                                        -Rob

                                        http://masterssummit.com

                                        http://qlikviewcookbook.com

                                          • Re: Governance_Dashboard - identify fields again ...
                                            Friedrich Hofmann

                                            Hi Rob!

                                             

                                            Well, then it probably has been changed because that's just where I looked ... the data has to be somewhere since the info is visible in the QEMC and there is a history that goes back to the last Server_restart I believe - so it should be possible to get it from somewhere ... there is the question whether it's worthwhile when it would be relatively easy to have each and every app just generate this 10kb file - but before thinking about that, I have to find where that info is ...

                                              • Re: Governance_Dashboard - identify fields again ...
                                                Friedrich Hofmann

                                                I guess I found it! At least I found some directories with data inside - I'll analyze what is there on the Train home and Report back tomorrow.

                                                Many thanks!

                                                  • Re: Governance_Dashboard - identify fields again ...
                                                    Friedrich Hofmann

                                                    Hi,

                                                     

                                                    there is one txt_file "Root" for each day and there is a Folder for each day with all the detailed Task_logs - but it seems I can get the exact´same info from either (I'm only looking for the Task_duration for now); So I guess I will go for that one txt_file per day, that's surely a lot less datacrunching ...

                                                      • Re: Governance_Dashboard - identify fields again ...
                                                        Friedrich Hofmann

                                                        Hey Rob,

                                                         

                                                        I have an issue with this table, I'm at a loss currently:

                                                        That table is quite okay, loading it is not a big Thing. The issue is somewhere else, only connected to that table. The Name of the table is "Monitoring".

                                                        I have a Loop "further down" in the script - which I already have in many other apps, it's in an INCLUDE file - basically parsing through the NoOfTables(), storing each one and filling a variable with all the table_names, separated by commas. Once the Loop has finished, that variable is cleansed and used to drop everything.

                                                        The issue is, the Moment the Loop Encounters that specific table, QlikView Exits the Group and immediately Exits the script altogether and it Fails.

                                                        Can you make head or tail of this? Is "Monitoring" some Kind of keyword for some easteregg or something?

                                                        Thanks a lot!

                                                        Best regards,

                                                         

                                                        DataNibbler

                                                          • Re: Governance_Dashboard - identify fields again ...
                                                            Tyler Waterfall

                                                            Freidrich - can you tell us what this comma-separated table list should look like (in order and with exact spelling of tables!).

                                                            Meaning - you expect the tables variable to be:

                                                            table1,table2,table3

                                                             

                                                            And question two - are you dropping tables as you add their name to this variable list?

                                                              • Re: Governance_Dashboard - identify fields again ...
                                                                Friedrich Hofmann

                                                                Hi Tyler,

                                                                 

                                                                I am not dropping the tables inside the Loop - that would dynamically Change the object_IDs, so it would not work.

                                                                The variable I build step by step finally Looks like

                                                                >>  '[table1], [table2], [table3], [table4]'  <<

                                                                That type of list is then used in the >> DROP TABLES << Statement. The order (left-to-right) follows the values of the Counter >> i <<. The spelling cannot be an issue since that variable is built inside the Loop, step by step, using the TableName() function ...

                                                                The table in question is loaded with the Name "Temp_Mon" and then renamed.

                                                                Also, that Loop is built into a great many other apps, too, and it works everywhere. Even now, it works - I have circumvented the Problem by 1) individually storing and dropping that particular table so the Loop never Encounters that one and 2) changing the Name - even after doing (1), I still had the same Thing Happening when the script encountered the STORE command and the tablename "Monitoring" - so, from a Logical Point of view, the issue could only have been the Name "Monitoring", though I cannot imagine how and why that should be an issue because I had a table labeled "Monitoring" before, just filled in a different way, and all was well ...