Scheduled reload does no reload on QS Enterprise cloud console (same app reloads maually correctly)

[Lars_SMS_to]

Dear all,

we are currently evaluation Qlik Sense Enterprise (on cloud), I have built an app that loads data (mainly from MySQL) and has the Dashboard visualizations in it. Every time I run the reload manually from the app's Data tab, it loads successfully (~10 minutes).

When I schedule this to run the reload automatically (Daily, Hourly, all the same), I see the scheduled time correctly.

After the set time, it also correctly jumps to showing the next scheduled time (e.g. tomorrow at the same time of the day if I use the daily trigger). There is no FAILED status or anything. Everything looks as if it was a successful reload.

But when I look at the details of the app in the hub the "last reload" timestamp is not updated and when I open the app and check the latest data, there was no new data loaded.

This video shows the problem: https://www.loom.com/share/7d2ab9ae1fb2431095dc0ca601cb29de

I have searched hours for any logs to help me understand what happened (or not happened) but I find nothing.

When I then manually reload again, it reloads successfully.

Any ideas from the community are very much appreciated!

Qlik support is not really helpful. They tell me the only way to get support's help for this is to open a case via the the support portal but I do not have and they cannot give me access to the portal since we're on an evaluation license. But How can I tell my CEO to make a financial commitment to buy licenses if (assuming it is a confirmed bug) this will mean I will have to manually reload my apps all the time and cannot use the scheduling functions.

[rzenere_avvale]

Goodmorning @Lars_SMS_to ,

I've tried on a QSE trial and it's working fine. My test application was really small (all the script was the one generated pressing ctrl+00).
How long does your app take to reload? Could you also check from an App level if the reload has been performed or not? You can create a text object with the following measure: =ReloadTime() . This will give you the last reload time, from an in-app perspective

Hope this helps,
Riccardo

[Lars_SMS_to]

Hey @rzenere_avvale ,
thanks for your reply.
Our app is not exactly small, it takes around 11 minutes to reload.
Will try your suggestion with =ReloadTime()

[rzenere_avvale]

Goodmorning @Lars_SMS_to,
then I would also suggest taking a look at the Event logs  related to start and event of a reload to see if the same timing is respected for scheduled reloads:

Riccardo

[squeakie_pig]

The SaaS scheduling options are ridiculously primitive.  It is unlikely this is a bug and something to do with your file. We have about 12 different apps that are scheduled to run throughout the day and they all reload fine.

If you are trialing the product here are some lessons learnt from our experiences that may be useful to you:

• If you need to reload something at a set time - you only have the option of setting it to run on the hour. 
• No task chaining - this only really becomes a problem because of ^^
• The event logs are very lean and often don't have failed error messages.
• The Personal > Shared > Managed state of Spaces doesn't work too well with established development processes
• No ability for performance scaling ie. sometimes it is sooo slow and you think "oh I'll just tweak this config setting / threshold... only to find NOPE ... doesn't exist in SaaS"  

I'm sure all these things will be updated and enhanced in the future, but for now it makes for a weak product.

As for your current issue, are you using include files?  Also, I found a file function not worky in SaaS. I think it was FileTime() and had to end up using QVDCreateTime() instead.  Maybe some of the other file functions are flakey.

[Lars_SMS_to]

Thanks again for both of your replies.

First regarding @rzenere_avvale s suggestions:
I scheduled it again and again the MMC has updated the Last execution timestamp but no new data was loaded.

1) I included ReloadTime() in the dashboard before scheduling the reload. After the reload it is not updated & still shows the time when I last reloaded manually

2) Event log is showing absolutely nothing for the time when I had this reload scheduled

Regarding @squeakie_pig  replies:

I agree 100% especially on the first 2 points. Even if the scheduled reload was working for me, those 2 points make it hardly possible to intelligently schedule tasks especially if they are depending on each other (create QVDs - then load from QVDs).

To answer your question:
I am doing pretty normal stuff in my load script, nothing complicated, no includes, no section access. I load data from the DB, I save and read QVDs and I use join and keep and mapping loads.

It's a young product and I don't like complaining. But these are vital functions and the Enterprise licenses are expensive. It is frustrating to see how well the actual tool fits our needs but the management functions are not flexible or even not working at all. And Qlik still does not seem to care about my issues.

[squeakie_pig]

This is going to be a pain for you to troubleshoot because at a minimum you need to wait ... EVERY HOUR to check to see if the schedule works!!!  Bwahahahahahah... Oh joy!  😞

When you say you manually reloaded, did you do it via the load script or via the elipses > Reload?  (Does it make a difference?)

Is anything else reloading via schedule (ie. different filelocations / different files / different space / different permissions / different data connectors)?   When you are running it as you, you might have access to the personal data connection. I didn't set up our environment so I don't think there is a Service Account.

Does running it in debug throw up any warnings or errors?

I don't know if you can export TRACE. If you could, it would essentially "create" a log file that might give more insight than the Events log.  Another option might be to add some kind of txt file store at different points in the script, that way you can actually see it has gotten to a certain point in the script.

Has the data actually changed in the data source?

Are you loading multiple files using an * in the filename? (I found out last week this didn't work on google docs with a spreadsheet)

[Lars_SMS_to]

Thanks, @squeakie_pig . Again very good ideas to isolate the problem.

Using the app's context menu on the hub seems to have the same result (reload is not done) like using the schedule function.

Here is what happens on the MMC log:

Spoiler

It creates an entry for “reload started” immediately after pressing reload on the app’s context menu:
2020-04-28T14:15:15Z	com.qlik/engine	com.qlik.app.reload.started	Lars Roettgers	
{
    "id": "a701bb98-003c-467d-bb40-f3bdad5d6623",
    "name": "SMSto messages",
    "reloadId": "5ea83a586b8d8500012a55fc",
    "spaceId": "",
    "startTime": "2020-04-28T14:15:15.743Z"
}

Then 4 minutes later another identical entry:
	
2020-04-28T14:19:33Z	com.qlik/engine	com.qlik.app.reload.started	Lars Roettgers	
{
    "id": "a701bb98-003c-467d-bb40-f3bdad5d6623",
    "name": "SMSto messages",
    "reloadId": "5ea83a586b8d8500012a55fc",
    "spaceId": "",
    "startTime": "2020-04-28T14:19:33.552Z"
}

The second entry was generated without me doing anything (I started the reload only once). And the first reload could not have been finished since reload time is 11-12 minutes.

Then the same happens again – again 4 minutes later:
2020-04-28T14:23:45Z	com.qlik/engine	com.qlik.app.reload.started	Lars Roettgers	
{
    "id": "a701bb98-003c-467d-bb40-f3bdad5d6623",
    "name": "SMSto messages",
    "reloadId": "5ea83a586b8d8500012a55fc",
    "spaceId": "",
    "startTime": "2020-04-28T14:23:45.123Z"
}

Then: no new entry, no “reload done” or similar. The app in the hub still says “updated 2 hours ago” (there was my last manual reload).
In the app’s details it says: “Last reload date: Apr 28, 2020 3:50 PM”.
ReloadTime() in the app gives me: “28/04/2020 12:50:44”.
Then I manually reloaded again by opening the app, going to the script and hit the “reload” button.
Again, one entry is created in the console log with a new reload_id:
2020-04-28T14:53:15Z	com.qlik/engine	com.qlik.app.reload.started	Lars Roettgers	
{
    "id": "a701bb98-003c-467d-bb40-f3bdad5d6623",
    "name": "SMSto messages",
    "reloadId": "d710690d-68c1-4f1a-aa76-1ae3b9dfdd98",
    "spaceId": "",
    "startTime": "2020-04-28T14:53:15.676Z"
}

But this time I also receive another log entry about the successful execution:
2020-04-28T15:09:48Z	com.qlik/engine	com.qlik.app.reload.ended	Lars Roettgers	
{
    "duration": 993121,
    "endTime": "2020-04-28T15:09:48.797Z",
    "errors": [],
    "id": "a701bb98-003c-467d-bb40-f3bdad5d6623",
    "name": "SMSto messages",
    "peakMemoryBytes": 5361817152,
    "reloadId": "d710690d-68c1-4f1a-aa76-1ae3b9dfdd98",
    "size": {
        "file": 0,
        "memory": 1660705227
    },
    "spaceId": "",
    "status": "ok",
    "warnings": []
}

ReloadTime() in the app now gives me: “28/04/2020 15:07:09”.

When I instead use the context menu's reload function on another app (a very simple one, where I just read to XLS files) it succeeds.
--> So something in my app seems to cause the problem.

To answer your other questions:

• Yes, I do use * LOADs to load several QVDs. Will try a modified version of my app with no *.
• Yes, the data is a live app so the data constantly changes. But this cannot be the problem since it works fine with manual reload and the data is always accurate after reload.
• About exporting TRACE: I have to investigate but I think it will not be possible. File handling seems very limited/difficult when using the Qlik cloud and I read several posts that many things are not possible or possible but will put your app in a "legacy" state, which I assume should be avoided.
• Debug mode: Not sure what you had in mind here. Can I schedule a debug load? Because yes, I can manually run the reload in debug but manually always runs without problems so this would not help me, I guess.

[rzenere_avvale]

Good afternoon @Lars_SMS_to ,

there is a "small" thing that is turning on a light: you have a "peakMemoryBytes": 5361817152 (which is 5,X GB)
As far as I know, and most likely I'm wrong on this point, there is a limit on the dimension of the application once it is in-memory, and for QSE is set to 2.5GB.
I don't know why you should be able to manually reload the application, but the dimension of the app seems a bit too much.

You should be able to ask Qlik Support what is the default in-memory dimension of the application, and if it is possible to expand it for your needs.
Meanwhile, you could also try to read less data inside that application, in order to be sure that it's only an issue with the actual size of the application and not something else.

I hope this helps,
Riccardo

[Lars_SMS_to]

Thanks! I am trying various changes to the load script to limit the file size, to not use * etc.

But just now I realized there is a reload log in the app's detail page.
It shows me this here: