23 Replies Latest reply: Oct 15, 2010 10:45 AM by Pablo Labbe RSS

    Tricks for scheduling dependent tasks

    Rob Wunderlich

      I'm looking for tips on scheduling large dependent task chains. In QVP 8.5, we had document sets which handled the problem nicely. But document sets are not available in V9.

      Here's the scenario. I have a group of 20 QVD reloads that need to execute first followed by the application reloads. The application reloads should not start unitil all QVD reloads are completed. The only way I can ensure all QVD reloads are complete is to run them serially and chain the application reloads off the last QVD.

      What I would like to do is run the QVDs in parallel and start the applications when all QVDs are complete. Any ideas on how to do this wihtout just having a single task chain for all QVDs?

      I''m facing this problem on both SBE edition and Publisher Enterprise.

      Thanks,
      -Rob

        • Tricks for scheduling dependent tasks

          Hi Rob,

           

          What about having a dependency on the QVW reload, that only start on success of the QVD reloads ?

            • Tricks for scheduling dependent tasks
              Vlad Gutkovsky

              Hi, Rob. Really interesting question. I'm not 100% sure, just throwing some ideas out there. Assuming you have a server with enough cores and memory, you can schedule up to 10 QVBs at the same time. So you could have the following structure: (i) schedule the first 10 QVD reloads to run at the same time, (ii) schedule the second 10 QVD reloads to run when #10 (or whichever is the longest) is completed; (iii) schedule the application reloads to start when #20 (or whichever is the longest in the second group) is completed. This would allow you to maximize parallel processing of the tasks.

              Does this help?

                • Tricks for scheduling dependent tasks
                  Rob Wunderlich

                  Hi Glad,

                  Thanks for the response. I don't think it will give me the required certainty. The applicaton reloads would run even if #15 -- somewhere in the middle of the pack -- had failed or not completed.

                  -Rob

                    • Tricks for scheduling dependent tasks
                      Vlad Gutkovsky

                      Ah, well to solve that you would use a combination of triggers and dependecies. The triggers would be set up as described above. Then you would make the applications dependent on tasks 1 - 20. This would ensure that they would only run if all QVDs reloaded successfully. Alternatively, if a certain application is dependent on less than all the QVDs, you could make fewer dependencies. This might be a little annoying to set up for each application, but it would give you the certainty you are looking for.

                      Cheers,

                        • Tricks for scheduling dependent tasks
                          Rob Wunderlich

                          I haven't found the Dependencies feature useful, perhaps I am misunderstanding it's application. A dependency seems to be satisified if the dependent task is in any state but "Failed". Therefore the dependent task runs if the dependency ran sucessfully in the past, or even if it never ran. I need some way to say "the dependency must have run today". Can I do that with dependencies?

                          -Rob

                  • Tricks for scheduling dependent tasks
                    Miguel �ngel Garc�a

                    Ok, here's an idea. Not tested really.

                    Make each of the 20 taks generate a txt file or write something somewhere. For Example Task A, at the end of the script, will write a txt File that says "Hey I already did my job", a counter of some sort.

                    That, in turn, (also at the end of the script) will execute a vbs that checks if the tasks have all done their job (count the number of files or something). If so, it will trigger an EDX for the final reload, if not it will wait for the next finished task to call it and check again.

                    OR, maybe you could read the task result logs, and apply the same logic.

                    Hope this helps. And please, let us know if it worked.

                    Mike.

                     

                     

                    • Tricks for scheduling dependent tasks
                      Pablo Labbe

                      What Qliktech has to say about this feature change ?

                      How it´s possible to migrate from one version to another when the newer version have missing features ?

                      • Tricks for scheduling dependent tasks
                        s j

                        Dear Rob,

                        One way to do this is make .bat file and write all task one by one in that batch file according to requirement.

                        Make QVW Application for QVD and then QVW.

                        Regards

                        Sunil Jain.

                         

                         

                         

                          • Tricks for scheduling dependent tasks
                            Paul Kelly

                            Utilising Windows Scheduler, I also use batch files to ensure that things run in the correct order....

                            Keep a spreadsheet which details what batch file a script / report is updated on.

                            I currently have 6 batch files...

                            Only have one small report scheduled using the QlikView Managemet Console and run this every 5 minutes to catch the update.

                            Would like to think that there is a better way than this...

                              • Tricks for scheduling dependent tasks

                                 

                                Hi,

                                Actually the Rube Goldberg machinery is not so complex. You need these:

                                • one database table called "rube" with two columns: qv_file (text) and counter (integer)
                                • one external command that does

                                del c:\wherever\*.qvd

                                • one database task that trigger that does

                                update [rube] set [counter]=0;

                                • for example 5 reload tasks to run in parallel (generating QVD files)
                                • one database task that does

                                update [rube] set [counter]=[counter]+1 where qv_file='rube';

                                • one QVW file with script bellow that will trigger the final reload file. This will wait for some time, than fail on timeout, or trigger the main task on success
                                • the main reload and publish task that will use the generated QVDs

                                 

                                 

                                Trigger the tasks as in the images bellow

                                 

                                 

                                CONNECT TO ..


                                for i=1 to 100

                                Counter:

                                REPLACE LOAD counter;

                                SQL SELECT counter FROM rube where qv_file='rube';

                                 

                                let c=peek('counter', 0, Counter);

                                 

                                trace $(c);

                                if $(c) = '5' then

                                exit script;

                                end if

                                 

                                sleep 1000 * 60;

                                next i


                                failure:

                                load *;

                                SQL select * from failure;

                                 

                                 

                                 

                                  • Tricks for scheduling dependent tasks
                                    Pablo Labbe

                                    Hi All, Qlikview 10 comes with a new described as follow:

                                     

                                     

                                    Trigger that fires when multiple events are completed

                                    In addition to the existing triggers which operate with OR logic when combined, we have

                                    added a new trigger with the possibility to combine the other triggers with AND logic.



                                     

                                    Did this feature solve your problem ?

                              • Tricks for scheduling dependent tasks
                                nikidaley

                                Hi Rob,

                                 

                                Did you ever find a solution to you dependent task problem ? I'm facing an almost identical scenario.

                                 

                                Rgds,

                                Nik

                                • Tricks for scheduling dependent tasks

                                   

                                  Hi,

                                   

                                  You can have an external command that deletes some flag files "del folder\flag*.qvd"

                                  "On successful execution" of this external command will fire the 20 reloads. Each will store a QVD file, and in the very end create a flag file flag_1.qvd with one row (content does not matter).

                                   

                                  One of the parallel will trigger on sucesful execution the main reload. This will count how many flag files there are, sleep and retry until there are enough. You do this a fixed amount of time, so there is no infinite loop.

                                   

                                  for i= 1 to 10

                                  set cnt=0;

                                  for each x in filelist('folder\flag*.qvd')

                                  set cnt = cnt + 1;

                                  next

                                   

                                  if cnt < 20

                                  sleep 1000 * 60 * 10; //10 minutes?

                                  //else

                                  //is there a short circuit "break" statement to quit the loop?

                                  end if

                                  next

                                   

                                  -Alex



                                  • Tricks for scheduling dependent tasks

                                    Rob,

                                     

                                    Did you ever get a resolution for your post about running app reloads after all 20 QVDs reloaded ? I am facing the same thing. We are on 8.5 and are moving to 9 but we have to rebuild our entire scheduling process because of this exact same thing....

                                      • Tricks for scheduling dependent tasks
                                        Rob Wunderlich

                                         


                                        pshicks wrote:Did you ever get a resolution for your post about running app reloads after all 20 QVDs reloaded ? I am facing the same thing. We are on 8.5 and are moving to 9 but we have to rebuild our entire scheduling process because of this exact same thing....


                                        No, I have not come up with a satisfactory solution. I'm stil running long task chains.

                                        -Rob