Runif not available for bigdata comonents

mamohan — Sat, 16 Nov 2024 06:03:21 GMT

Hi,

I would like to know why the bigdata components don't have runif trigger. I have tried to use a tjava component and checked if it has runif. It didnt work. I would like to trigger joblets from a bigdata job based on the inputs from context variables.

Re: Runif not available for bigdata comonents

Anonymous — Thu, 18 Apr 2019 08:16:10 GMT

Hello,

RunIf and OnComponentOK triggers are not supported in Big Data job designers.

This limitation is valid for M/R and Spark jobs and is due to the way Big Data jobs are working and due to their distributed context : Big Data execution engines evaluate the Subjobs as a whole and not by component like Standard/DI job is doing, meaning that several components can be merged and processed in the same step by Spark (ex: if you do tExtractJson->tFilterCols->tFilterRow, this will be executed by Spark as one single map operation). As such, we can not guarantee that an individual component respects RunIf / OnComponentOK triggers (as this is not the granularity in the hand of the execution engine), and so these 2 triggers are not active in these job designers.
Please have a look at this related jira link:https://jira.talendforge.org/browse/TBD-4401

Best regards

Sabrina

topic Re: Runif not available for bigdata comonents in Talend Studio

Runif not available for bigdata comonents

Re: Runif not available for bigdata comonents