Suggest an Idea

[sriniK]

Compose Data warehouse workflow does not allow running multiple ETL set in parallel. It keeps throwing error - 

• Task SRC_XREF and task SRC_DRM_FINANCE cannot be run in parallel as at least one of the source entities in their mappings is the same.
• Task SRC_DRM_FINANCE and task SRC_XREF cannot be run in parallel as at least one of the source entities in their mappings is the same.
• Task SAPS4HANA_SAPS4_ST_LR_Z3 and task SRC_DRM_FINANCE cannot be run in parallel as at least one of the source entities in their mappings is the same.

Currently we have 15 ETL sets and it takes 5.30 hours to complete the warehouse load. If its possible to run the ETL sets parallelly then  the run time can come down considerably.

[TimGarrod]

Hi @sriniK , 

In Compose data warehouse projects, we currently don't allow for multiple parallel ETL  sets when the same target tables are in a task.   This is to prevent duplicate loading of data (if 2 tasks to load TABLEA are running in parallel, Compose could inadvertently load the same business / natural key to a target table).   This appears to be the case in your 1st instance. 

Typically, customers will put all mappings that load the same target structures into the same ETL set.  This ensures parallel loading of the mappings (even across sources) to specific target entities without concern for duplicate data being loaded to the vault. 

Can you confirm that the above scenario is what you are encountering with your scenario (different mappings loading to the same target tables ) ? 

Can you 

[ubanerjee]

Compose errors out an ETLSet if another is already running. Would it make sense to keep the scheduled ETLset as pending and run it once the currently running job is over? Workflows allow some of this, but jobs that need to start at fixed times fail if the previous job runs for an extended period.

[Meghann_MacDonald]

From now on, please track this idea from the Ideation portal. 

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.