Pattern for tracking processing status in the database

FabianB — Fri, 15 Nov 2024 23:57:37 GMT

Hi all,

I have a table with some data and a status like this:

A B Status

Data1 Data2 New

I read this data with a Db2Input and then write data elsewhere with a Db2Output.

Now I want to set the Status to Processed if I can successfully process and write the data or set the status to Error (and possibly the cause of the error in a separate field) for those that have failed.

A B Status

Data1 Data2 Processed

Since I have many records (> 1 Mio), I use DBoutput in Bulk mode.

Can someone recomend a pattern for a workflow that handles this scenario?

In single line processing mode (Db2Output without batch size) it should be easy, but the performance is not sufficient.
I should probably do all kind of validations beforehand and not attempt to write the data and instead update the status to error. a db error should be not part of a normal processing scenario (?)

If someone has a pattern or example I would be grateful.

Thanks

Fabian

Re: Pattern for tracking processing status in the database

Anonymous — Thu, 22 Jul 2021 07:58:39 GMT

As the reject flow option can't used with Bulk mode together, I would like do the validations beforehand using tSchemaCompilanceCheck component and get the reject rows, these rows will be updated the status.

Regards

Shong

topic Re: Pattern for tracking processing status in the database in Talend Studio

Pattern for tracking processing status in the database

Re: Pattern for tracking processing status in the database