Qliksense Reload background activites

[Anonymous]

What all activities happens in background when reload is triggered through Qliksense scheduler? looking for step wise detailed explanation.

[jwjackso]

Someone else can probably provide definitive information, but this is what I have observed for our tasks.  For a successful  task, the execution results are:

Changing task state to Triggered.

Trying to start task.  Sending task to slave scheduler

Changing task state from Triggered to Started

Reference to scriptlog added

Changing task state from Started to FinishedSuccess

The possible execution statuses are below:

0,"Never Started"

1,"Triggered"

2,"Started"

3,"Queued"

4,"Abort Initiated"

5,"Aborting"

6,"Aborted"

7,"Finished Success"

8,"Finished Failed"

9,"Skipped"

10,"Retry"

11,"Error"

12,"Reset"

[Levi_Turner]

At a first approximation based on knowledge and a few hunches:

• The task is triggered at the Repository level
• The Repository determines which nodes have the app load balanced to it
• It asks the nodes with the app who have slave schedulers which node is least busy
• Once it receives a response, it allocates the reload job to the least busy scheduler node
• The slave scheduler polls the Engine to determine whether it has a free slot for it determined by the Max Concurrent Reload setting on the Scheduler
• If the Engine has a slot, free, then it signals to the Engine that it needs to reload the app
• The Engine begins to load the app and load script to execute the script
• A script log is written to %ProgramData%\Qlik\Sense\Log\Script for the reload
• The App is reloaded as defined in the load script
• Progress is polled to the Slave scheduler
• When the script execution is finished, the Engine saves the diff in the Apps > Transactions folder then coordinates writing of the app based on the availability of a file lock
• Once the app is saved, the Engine either leaves the App in memory if it was there before or unloads it from memory if it wasn't
• The engine signals to the Scheduler that its execution has finished
• The scheduler signals to the Repository that the app has been reloaded
• The script log is moved to the ArchivedLog\Node\Script path
• The reload job is signaled as successful to the repository

[martinsandin]

I'll chime in here from the Scheduler perspective:

1. The task is triggered in the Master Scheduler, either by call from the Repository or by a schedule or chain trigger
2. Unless the task is already running the Master marks it as 'Triggered'
3. The Master Scheduler asks each slave if they can run the task and then sends it to the one which and has the lowest CPU load (the Slaves will check their Engine and Repository connections during this process)
4. If the Slave is already using all it's engine connections it marks the task as 'Queued' and puts it in the Queue until there is a free Engine connection
5. When there is a free engine connection the Slave signals the Master to mark the task as 'Started' and signals the Engine to start the reload
6. During the reload process the Slave poll progress from the Engine
7. If the engine finishes successfully the Slave will tell the Engine to save the App
8. The Slave tells the Repository to archive the script execution log
9. The Slave signals the Master that the task is finished successfully or as a failure
10. If the task failed it may now be retried in which case the Master marks it as 'Retry' and sends it back to step (3)
11. The Master marks the execution as 'FinishedSuccess' or 'FinishedFail'

On top of this there is various levels of timeouts, internal details of the Engine, and various entities the Master scheduler creates, updates, and removes from the Repository (ExecutionSession, ExecutionResult, ...).

Besides this basic flow there are also other states which will be entered

• Abort-states can be entered manually from the QMC or by the various timeouts the Master handles
• 'Reset' can be entered at Master Scheduler startup if the task is loaded and found in a state which can't be otherwise handled
• 'Skipped' which happens when the task is triggered but the Task or App are disabled
• 'Error' if the execution finishes because no Slave was found, because of Master communication errors, or for unknown internal reasons

[konidena]

Hi Turner,

You mentioned the below point from the list.

• When the script execution is finished, the Engine saves the diff in the Apps > Transactions folder then coordinates writing of the app based on the availability of a file lock

What is "Saves the Diff" in the sentence. Could you please explain. what will save in Transactions folder.

Regards

Srinivas