Send an e-mail when a job is started with rows count

[s_mil]

Hi everybody!

I've created my first pipeline with Talend Studio. This is a really simple flow, where I get some data from a web service, i transform that data before making an upsert to a Microsoft SQL database.

The image below is my current pipeline:

 

I've highlighted the tNormalize_1 step because this is the step where I transform my data from a unique string to a proper object with about 93k rows.

Since I've seen that the flow takes about 1h30m to be completed with the final tDBOutput_1 step which makes the upsert to a database, I thought to add a tSendMail component when the data is normalized because I would like to receive an e-mail that is telling me the process is started (with date time) and how many records are going to be processed.

Clearly, what I've done doesn't work because component tNormalize returns "ok status" when the whole process is completed. The tNormalize doesn't transform and pass massively the object to the next step, but it passes about 30 rows per second.It is probably a very silly question and problem, but how can I send an e-mail containing the data I explained before when a job is started?

I also add below an image of the running process:

 

Thanks.

[Prakhar1]

Can you add one more tJavaFlex after the tNormalize.

If yes then in the main code of javaflex write

context.count++;

and then create a link of onComponentOK to the tSendEmail.

Then use the count context to send the count of the rows generate by tNormalize.

Also in the context declaration tab, initialize count to 0 as Integer data type.

[s_mil]

Hi @prakhar dubey​ ! Thanks for your reply.

I've modified the pipeline as you can see in the image below:

I receive an e-mail with the correct number of rows, but this happens when the flow is completed and not when the process has started.

I would like to receive this e-mail when the job starts because the insert on MS SQL database takes some time, and I'll receive another e-mail when the insert process is over.

[Prakhar1]

@Stefano Milziadi​  you mean you receive the count email when the rows are inserted into the MsSQL, right ?

This should not happen ideally because as soon as Java Flex count the rows it will send and email before the insertion operation, but if this is not happening please let me i will thing of something else.

[s_mil]

@prakhar dubey​ With the pipeline above, I receive one e-mail (from tSendMail_4) when all inserts made by tDBOutput_1 are completed. So this is not useful because I receive at the same time the e-mail from tSendMail_4 and the one from tSendMail_1/3. I don't know if I'm doing something wrong in configuration (sorry, this is my first pipeline) or your suggested solution is not working in this case.

Actually, I tried what you suggested before because this is the same way I thought it has to work, but I've noticed that not all rows are passed from tNormalize_1 to the next step in one time, but the rows are passed in chunks based on SQL inserts made (look at the second image in the first post, 495 rows on 93k total).

Let me know if you have something else in mind. Thanks 🙂

[Prakhar1 · Accepted solution]

@Stefano Milziadi​ yes i tried it by myself also and facing the same issue when using a java component in the flow not with other output components.

I don't know if you like the new solution or not but what you can do,

after JavaFlex use tBufferOutput, then create a onSubJobOk link from tWebService and then use tBufferInput and then attach the next flow from tBufferInput.

just like I did and in place of tJava_1 there will be the email component.

[s_mil]

Hi @prakhar dubey​ , I've changed a bit your flow to meet my needs, and it seems working fine. In my case I've attached the "onComponentOk" of tBufferOutput_1 to the tBufferInput_1, in this way I don't have to replicate the tNormalize component.

This is the "final" result (so far):

 

Thanks again for your help, really appreciated 🙂

[Prakhar1]

@Stefano Milziadi​ thank you too, i m glad to help you here.

Please select it as a solution and a kudos will be really appreciated 🙂