Skip to main content

Official Support Articles

Search or browse our knowledge base to find answers to your questions ranging from account questions to troubleshooting error messages. The content is curated and updated by our global Support team

Announcements
CUSTOMERS ONLY: Now accepting customer applications for the 2023 Luminary Program: SUBMIT NOW

How to deal with Out of Memory errors in an automation (QAA)

cancel
Showing results for 
Search instead for 
Did you mean: 
AfeefaTk
Support
Support

How to deal with Out of Memory errors in an automation (QAA)

In this article, we will go through possible ways to recognize out-of-memory errors that can happen within automation, the root cause, and best practices to avoid/get rid of memory issues.

How to recognize

Out of memory errors can be identified if any of the below occurs

  • The automation fails with the 'Execution has failed' error message
  • The maximum run duration of automation takes much longer than the normal execution time and exceeds the limit which results in automation failure

How to debug

  • Maximum memory usage during an automation run is 256 MB
  • Download job history by clicking the 'Export this run' button and check if memory usage accumulative sum exceeds the above limit.

Export-Job-RunExport-Job-Run

Root Cause

The automation runs out of memory if it has the following blocks with huge data or dealing with larger lists

Best Practices

Let's walk through different ways of resolving memory issues by recreating the automation / changing the behaviour of automation

  • In contact sync or any data sync use-cases where automation is used to sync data between 2 CRM systems and if the account contains a huge amount of data(>1million), the best approach is to use backfill patterns. Refer: https://community.qlik.com/t5/Knowledge/Backfill-patterns/ta-p/1806691 
  • Split automation
    • If Multiple syncs happen in one single automation which means for eg syncing contacts, accounts, leads, etc within the same automation, its always recommended to split the automation into 3 separate automation which will be of the same pattern as described below
      • Sync contacts from source to destination CRM in 1st automation
      • Sync accounts from source to destination CRM in 2nd automation
      • Sync leads from source to destination CRM in 3rd automation etc...
    • If merge lists block is used to merge two given lists (arrays) into one new list where any/both of the lists have more data, replace the merge list block and perform data sync for individual lists in separate automation as described in the above step
  • If compare lists block is used to delete records from destination that are missing from the source,

    automation may fail with an out of memory error for very large lists. So in order to make the deletion process work we can change the behaviour of the automation. Instead of deleting records based on the comparison, we will implement automation in any other possible ways described below

  • Keep in mind the below points while using variables in an automation
    • Do not create a copy of the variable which is having larger dataset if it is not required.
    • Do not use variables to store the last output of a block. The last output of a block is always kept in memory and can be used in the input field of other blocks
    • Do not store output from list endpoints to a variable. In order to get all data loop through the data using the same list endpoint or use the Loop block and then use it as an input in subsequent blocks

Related Resources:

Labels (2)
Version history
Last update:
‎2022-05-23 05:25 AM
Updated by:
Contributors