idea Re: Running data warehouse ETL sets in Parallel - Status changed to: Open - New in Suggest an Idea

Running data warehouse ETL sets in Parallel

sriniK — Fri, 13 Aug 2021 19:33:39 GMT

Compose Data warehouse workflow does not allow running multiple ETL set in parallel. It keeps throwing error -

Task SRC_XREF and task SRC_DRM_FINANCE cannot be run in parallel as at least one of the source entities in their mappings is the same.
Task SRC_DRM_FINANCE and task SRC_XREF cannot be run in parallel as at least one of the source entities in their mappings is the same.
Task SAPS4HANA_SAPS4_ST_LR_Z3 and task SRC_DRM_FINANCE cannot be run in parallel as at least one of the source entities in their mappings is the same.

Currently we have 15 ETL sets and it takes 5.30 hours to complete the warehouse load. If its possible to run the ETL sets parallelly then the run time can come down considerably.

Re: Running data warehouse ETL sets in Parallel - Status changed to: Open - New

TimGarrod — Mon, 10 Jan 2022 23:17:53 GMT

Hi @sriniK ,

In Compose data warehouse projects, we currently don't allow for multiple parallel ETL sets when the same target tables are in a task. This is to prevent duplicate loading of data (if 2 tasks to load TABLEA are running in parallel, Compose could inadvertently load the same business / natural key to a target table). This appears to be the case in your 1st instance.

Typically, customers will put all mappings that load the same target structures into the same ETL set. This ensures parallel loading of the mappings (even across sources) to specific target entities without concern for duplicate data being loaded to the vault.

Can you confirm that the above scenario is what you are encountering with your scenario (different mappings loading to the same target tables ) ?

Can you

Re: Running data warehouse ETL sets in Parallel - Status changed to: Open - Collecting Feedback

TimGarrod — Mon, 14 Mar 2022 19:43:28 GMT

Re: Running data warehouse ETL sets in Parallel

ubanerjee — Tue, 12 Apr 2022 21:55:11 GMT

Compose errors out an ETLSet if another is already running. Would it make sense to keep the scheduled ETLset as pending and run it once the currently running job is over? Workflows allow some of this, but jobs that need to start at fixed times fail if the previous job runs for an extended period.

From now on, please track this idea from the Ideation por...

Meghann_MacDonald — Tue, 05 Sep 2023 13:13:48 GMT

From now on, please track this idea from the Ideation portal.

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Re: Running data warehouse ETL sets in Parallel - Status changed to: Closed - Archived

Ideation — Tue, 05 Sep 2023 13:13:50 GMT