How to deal with Out of Memory errors in an automation (QAA)

[AfeefaTk]

In this article, we will go through possible ways to recognize out-of-memory errors that can happen within automation, the root cause, and best practices to avoid/get rid of memory issues.

How to recognize

Out of memory errors can be identified if any of the below occurs

• The automation fails with the 'Execution has failed' error message
• The maximum run duration of automation takes much longer than the normal execution time and exceeds the limit which results in automation failure

How to debug

• Maximum memory usage during an automation run is 256 MB

• Download job history by clicking the 'Export this run' button and check if memory usage accumulative sum exceeds the above limit.

Export-Job-Run

Root Cause

The automation runs out of memory if it has the following blocks with huge data or dealing with larger lists

• List endpoints that may have a huge amount of data(>1million) 
• Merge list blocks. Refer:https://help.qlik.com/en-US/blendr/Content/blend-editor/merge-lists-block.htm
• Compare list blocks. Refer:https://help.qlik.com/en-US/blendr/Content/blend-editor/compare-lists-block.htm
• Deduplicate list blocks. Refer:https://help.qlik.com/en-US/blendr/Content/blend-editor/deduplicate-list-block.htm
• Excessive usage of variables which is having larger dataset .

Best Practices

Let's walk through different ways of resolving memory issues by recreating the automation / changing the behaviour of automation

• In contact sync or any data sync use-cases where automation is used to sync data between 2 CRM systems and if the account contains a huge amount of data(>1million), the best approach is to use backfill patterns. Refer: https://community.qlik.com/t5/Knowledge/Backfill-patterns/ta-p/1806691 
• Split automation
  • If Multiple syncs happen in one single automation which means for eg syncing contacts, accounts, leads, etc within the same automation, its always recommended to split the automation into 3 separate automation which will be of the same pattern as described below
    • Sync contacts from source to destination CRM in 1st automation
    • Sync accounts from source to destination CRM in 2nd automation
    • Sync leads from source to destination CRM in 3rd automation etc...
  • If merge lists block is used to merge two given lists (arrays) into one new list where any/both of the lists have more data, replace the merge list block and perform data sync for individual lists in separate automation as described in the above step
• If compare lists block is used to delete records from destination that are missing from the source,

  automation may fail with an out of memory error for very large lists. So in order to make the deletion process work we can change the behaviour of the automation. Instead of deleting records based on the comparison, we will implement automation in any other possible ways described below

  • Check if there is already an On delete webhook event from the source platform, to delete records in the destination. Please get in touch with our support team if a connector is missing webhooks (that should be available according to the API documentation). Refer: https://help.qlik.com/en-US/blendr/Content/design-patterns/design-patterns-to-delete-records.htm
• Keep in mind the below points while using variables in an automation
  • Do not create a copy of the variable which is having larger dataset if it is not required.
  • Do not use variables to store the last output of a block. The last output of a block is always kept in memory and can be used in the input field of other blocks
  • Do not store output from list endpoints to a variable. In order to get all data loop through the data using the same list endpoint or use the Loop block and then use it as an input in subsequent blocks

Related Resources:

• Qlik Application Automation for OEM platform limits 

[_Anders_]

My cause to this issue is trying to send multiple pdf-reports. Trying to move customers from on prem installations with Nprinting to Saas is not easy. I hit this limit at around 60 reports and the only way around it I found is using copies of the same automation and using the call automation block in combination with a distinct field value or variable to separate the reports in the different automations. Not a practical solutions when I need to send out 300-500 different versions of the same report. How would you go about doing this? We need a better option if we're going to be able to move Nprinting customers to Saas.  

[AfeefaTk]

Hi @_Anders_ 

Could you please share the automation json file which failed for you?

What error did you get?

Is it an Execution has failed error?

Thanks

[_Anders_]

Yea, please have a look @AfeefaTk . I put the automation in a dropbox. I have a log as well but I don't want to share this openly since it contains e-mail addresses and names.  

I create a two side pdf report and use get straight table data in two loops, first selecting a person and then going through all customers connected to this person generating a report for every customer, before going to the next person. I usually get out of memory around report 60, so I tried splitting this into two automations but it does not seem practical since I now hit the ceiling again and need to split it into three automations. And I now face a similar task but with more than 300 reports need to be sent. 

[AfeefaTk]

Hi @_Anders_ 

Is it possible for you to share the automation json file and also the job run id?

Thanks

[_Anders_]

@AfeefaTk So I did som tests and it seems it's the send mail box that takes up most memory capacity. Running the reports and saving them to OneDrive I could do without problem for 120 reports, but instead of saving them sending them by mail only got me to report 58 before the job ran out of memory. 

Automation id that failed is  cc218dc0-ce26-11ed-ac38-27f606065a5b and the run id 45542550-ce27-11ed-82a5-d505a652cb1f. 

Link to automation 

[AfeefaTk]

Hi @_Anders_ 

We are investigating this issue.

Please allow us some time and we will get back to you as soon as we have an update.

Meanwhile could you please let us know if you have already opened a salesforce case?

If yes, could you please share the salesforce case number?

Thanks

[_Anders_]

@AfeefaTk I have made no other report on this issue. 

[_Anders_]

I can confirm that a fix released today solved this issue. At least for up to 120 reports sent by mail in one automation witch is the most I tired so far.  

[paulcalvet]

Hello @AfeefaTk ,

I have the same problem with the bloxk export app.

I try to backup all applications in shared space, the ticket number is 00094666: Automation failed without error

Can you help me ? 

Thanks,

Paul

[Sonja_Bauernfeind]

Hello @paulcalvet 

Your ticket is being actively worked on and our support engineers will be assisting you in accordance with our SLAs. We will collaborate internally to get the expertise we need.

All the best,
Sonja