topic Re: Send an e-mail when a job is started with rows count in Talend Studio

Send an e-mail when a job is started with rows count

s_mil — Sat, 16 Nov 2024 00:53:33 GMT

Hi everybody!

I've created my first pipeline with Talend Studio. This is a really simple flow, where I get some data from a web service, i transform that data before making an upsert to a Microsoft SQL database.

The image below is my current pipeline:

I've highlighted the tNormalize_1 step because this is the step where I transform my data from a unique string to a proper object with about 93k rows.

Since I've seen that the flow takes about 1h30m to be completed with the final tDBOutput_1 step which makes the upsert to a database, I thought to add a tSendMail component when the data is normalized because I would like to receive an e-mail that is telling me the process is started (with date time) and how many records are going to be processed.

Clearly, what I've done doesn't work because component tNormalize returns "ok status" when the whole process is completed. The tNormalize doesn't transform and pass massively the object to the next step, but it passes about 30 rows per second.It is probably a very silly question and problem, but how can I send an e-mail containing the data I explained before when a job is started?

I also add below an image of the running process:

Thanks.

Re: Send an e-mail when a job is started with rows count

Prakhar1 — Tue, 29 Dec 2020 14:20:40 GMT

Can you add one more tJavaFlex after the tNormalize.

If yes then in the main code of javaflex write

context.count++;

and then create a link of onComponentOK to the tSendEmail.

Then use the count context to send the count of the rows generate by tNormalize.

Also in the context declaration tab, initialize count to 0 as Integer data type.

Re: Send an e-mail when a job is started with rows count

s_mil — Wed, 30 Dec 2020 08:08:29 GMT

Hi @prakhar dubey ! Thanks for your reply.

I've modified the pipeline as you can see in the image below:

I receive an e-mail with the correct number of rows, but this happens when the flow is completed and not when the process has started.

I would like to receive this e-mail when the job starts because the insert on MS SQL database takes some time, and I'll receive another e-mail when the insert process is over.

Re: Send an e-mail when a job is started with rows count

Prakhar1 — Wed, 30 Dec 2020 09:04:48 GMT

@Stefano Milziadi you mean you receive the count email when the rows are inserted into the MsSQL, right ?

This should not happen ideally because as soon as Java Flex count the rows it will send and email before the insertion operation, but if this is not happening please let me i will thing of something else.

Re: Send an e-mail when a job is started with rows count

s_mil — Wed, 30 Dec 2020 09:14:56 GMT

@prakhar dubey With the pipeline above, I receive one e-mail (from tSendMail_4) when all inserts made by tDBOutput_1 are completed. So this is not useful because I receive at the same time the e-mail from tSendMail_4 and the one from tSendMail_1/3. I don't know if I'm doing something wrong in configuration (sorry, this is my first pipeline) or your suggested solution is not working in this case.

Actually, I tried what you suggested before because this is the same way I thought it has to work, but I've noticed that not all rows are passed from tNormalize_1 to the next step in one time, but the rows are passed in chunks based on SQL inserts made (look at the second image in the first post, 495 rows on 93k total).

Let me know if you have something else in mind. Thanks 🙂

Re: Send an e-mail when a job is started with rows count

Prakhar1 — Wed, 30 Dec 2020 13:13:56 GMT

@Stefano Milziadi yes i tried it by myself also and facing the same issue when using a java component in the flow not with other output components.

I don't know if you like the new solution or not but what you can do,

after JavaFlex use tBufferOutput, then create a onSubJobOk link from tWebService and then use tBufferInput and then attach the next flow from tBufferInput.

just like I did and in place of tJava_1 there will be the email component.

Re: Send an e-mail when a job is started with rows count

s_mil — Thu, 31 Dec 2020 08:02:09 GMT

Hi @prakhar dubey , I've changed a bit your flow to meet my needs, and it seems working fine. In my case I've attached the "onComponentOk" of tBufferOutput_1 to the tBufferInput_1, in this way I don't have to replicate the tNormalize component.

This is the "final" result (so far):

Thanks again for your help, really appreciated 🙂

Re: Send an e-mail when a job is started with rows count

Prakhar1 — Thu, 31 Dec 2020 08:24:39 GMT

@Stefano Milziadi thank you too, i m glad to help you here.

Please select it as a solution and a kudos will be really appreciated 🙂