I have a job that calls/triggers close to 20 child jobs in each iteration...by design the job should go to next iteration only if all the jobs are successfully completed...(however i do not want to enable die on error) For audit and reconciliation purposes i need to capture stats at component level as well as job level, if i go the route of using tFixedFlowInput and start capturing all the after flow variables it will be too much of overhead on the process as i am already exploiting the available memory to its peaks (using all kinds of hash,cache and buffer components). Other than using tStatCatcher is there any other way of capturing the stats. In some ETL tools we have an option to get the status of the job seamlessly, is there any such way to capture the job execution status in Talend.
This is a stream of data that originates from a Queue, because of a limitation in Talend(Code limit 65535 bytes) I am forced to read the source data again and again, so to avoid that i am carrying the source data end-to-end through internal memory.
Coming to hand coding of the flow variables, i think it is not a great way to capture Job status because i have close to 28 jobs that gets called in the interim, for each job if i take 5 variables (no. of records sourced, no. of inserts, updates,rejects and deletes) then i am looking at 140 variables that i end up hand coding, i feel there will be a better way to handle this.
In my view this is a very generic use case and pretty much every implementation would have faced this situation...can some one please share some examples or approaches as to how they dealt with this requirement.
