In this article, we will go through possible ways to recognize out-of-memory errors that can happen within automation, the root cause, and best practices to avoid/get rid of memory issues.
How to recognize
Out of memory errors can be identified if any of the below occurs
- The automation fails with the 'Execution has failed' error message
- The maximum run duration of automation takes much longer than the normal execution time and exceeds the limit which results in automation failure
How to debug
- Maximum memory usage during an automation run is 256 MB
-
Download job history by clicking the 'Export this run' button and check if memory usage accumulative sum exceeds the above limit.
Root Cause
The automation runs out of memory if it has the following blocks with huge data or dealing with larger lists
- List endpoints that may have a huge amount of data(>1million)
- Merge list blocks. Refer:https://help.qlik.com/en-US/blendr/Content/blend-editor/merge-lists-block.htm
- Compare list blocks. Refer:https://help.qlik.com/en-US/blendr/Content/blend-editor/compare-lists-block.htm
- Deduplicate list blocks. Refer:https://help.qlik.com/en-US/blendr/Content/blend-editor/deduplicate-list-block.htm
- Excessive usage of variables which is having larger dataset .
Best Practices
Let's walk through different ways of resolving memory issues by recreating the automation / changing the behaviour of automation
- In contact sync or any data sync use-cases where automation is used to sync data between 2 CRM systems and if the account contains a huge amount of data(>1million), the best approach is to use backfill patterns. Refer: https://community.qlik.com/t5/Knowledge/Backfill-patterns/ta-p/1806691
- Split automation
- If Multiple syncs happen in one single automation which means for eg syncing contacts, accounts, leads, etc within the same automation, its always recommended to split the automation into 3 separate automation which will be of the same pattern as described below
- Sync contacts from source to destination CRM in 1st automation
- Sync accounts from source to destination CRM in 2nd automation
- Sync leads from source to destination CRM in 3rd automation etc...
- If merge lists block is used to merge two given lists (arrays) into one new list where any/both of the lists have more data, replace the merge list block and perform data sync for individual lists in separate automation as described in the above step
- If Multiple syncs happen in one single automation which means for eg syncing contacts, accounts, leads, etc within the same automation, its always recommended to split the automation into 3 separate automation which will be of the same pattern as described below
- If compare lists block is used to delete records from destination that are missing from the source,
automation may fail with an
out of memory
error for very large lists. So in order to make the deletion process work we can change the behaviour of the automation. Instead of deleting records based on the comparison, we will implement automation in any other possible ways described below- Check if there is already an
On delete
webhook event from the source platform, to delete records in the destination. Please get in touch with our support team if a connector is missing webhooks (that should be available according to the API documentation). Refer: https://help.qlik.com/en-US/blendr/Content/design-patterns/design-patterns-to-delete-records.htm
- Check if there is already an
- Keep in mind the below points while using variables in an automation
- Do not create a copy of the variable which is having larger dataset if it is not required.
- Do not use variables to store the last output of a block. The last output of a block is always kept in memory and can be used in the input field of other blocks
- Do not store output from list endpoints to a variable. In order to get all data loop through the data using the same list endpoint or use the Loop block and then use it as an input in subsequent blocks
Related Resources:
Yea, please have a look @AfeefaTk . I put the automation in a dropbox. I have a log as well but I don't want to share this openly since it contains e-mail addresses and names.
I create a two side pdf report and use get straight table data in two loops, first selecting a person and then going through all customers connected to this person generating a report for every customer, before going to the next person. I usually get out of memory around report 60, so I tried splitting this into two automations but it does not seem practical since I now hit the ceiling again and need to split it into three automations. And I now face a similar task but with more than 300 reports need to be sent.
Hello @AfeefaTk ,
I have the same problem with the bloxk export app.
I try to backup all applications in shared space, the ticket number is 00094666: Automation failed without error
Can you help me ?
Thanks,
Paul
