How to perform a backfill using parallel webhooks

[Emile_Koslowski]

This article explains how a backfill sync can be built with webhook blends that run in parallel.

More information about webhooks can be found in this article under "Automated Webhooks".

This backfill sync leverages the parallel execution capabilities of webhooks. To divide the total amount of records that need to be synced over multiple webhooks.

This sync will be built as a bundle with one main blend "the orchestrator" and X amount of webhook blends. The example in this article will use 2 webhook blends to process records.

In this article, the placeholder {bundleguid} is used. When a blend is executed, this will resolve to guide of that bundle's unique installation. More information on placeholders can be found here.

Design of the bundle

The main blend (orchestrator) will fetch records from a source and send them to the CDP. Each webhook blend is listening to new & updated records in the CDP and will process them one by one and send them to the destination system.

Orchestrator

• Fetches records from source (eg. Shopify)
• Loops over them in batch
  • Each batch is sent to the CDP with the block "Bulk Upsert Custom Objects"
• The "Bulk Upsert Custom Object - CDP" block will use different scopes to specify which webhook blend should process the batch.
  • For this sync, two scopes are used, "{bundleguid}_1" and "{bundleguid}_2".
  • Which scope the batch update should happen to is specified by the variable "Counter".
  • When a batch is upserted to "{bundleguid}_2", the counter is reset.

Webhooks

• Created by configuring the Start block to run in "Webhook" run mode
• Will run when a new record is added/updated in the CDP
• Each CDP webhook listens only to new/updated records in a specific scope (see settings tab in the Start block)
• After each record is processed (sent to the target platform), the record should be deleted from the CDP
• Keep our platform limits in mind when deciding how many records each webhook should process

General remarks

1. Keep the webhook blend simple, in this example it executes an upsert snippet. This snippet will first execute a get block in the target system (Splio) and then, if it found a record, it will execute an update block, else it will execute a create record block.
2. Limit the number of webhook blends to 5 blends. 
3. If something goes wrong in a webhook blend, the record won't be deleted from the CDP. When the sync is finished, another blend can be used to retry the records that are still in the CDP.

Results

Webhooks	Sleep time	Total process time	Total records	Time per record
None, regular approach with 1 blend that does everything.	NA	661 seconds	1106	0,59765 seconds
2	2 seconds	132 seconds	1106	0,12115 seconds
3	2 seconds	116 seconds	1106	0,10488 seconds
5	2 seconds	86 seconds	1106	0,07504 seconds

* The above results vary depending on the webhook blends complexity and the actual data that makes up the records.

Attached to this article are the used orchestrator blend and an example of a webhook blend. More information on importing blends from a JSON file can be found here.

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.