Update a csv field

[Anonymous]

Hello everyone,

In order to count how many times my job had an error I want to create a csv file that would be updated each time there is an error in my job.

My job just sends files using ftp connection it has a tftpfilelist ==> tftpget ==> tftpput.

My problem is that I have to count the error for each file separately for example before the execution my file would look like this :

file1;0

file2;0

file3;0

after execution if there is a problem on file 2 I get :

file1;0

file2;1

file3;0

and I can't figure out a way to update just that value.

PS : If you have a better idea than using a .csv file I am open to suggestion 

[tnewbie]

Hi DjallelB,

some ETL tools allow file updates but, I seriously doubt if Talend allows a file update. If you are very keen on going the file route and if you have Oracle access, you can create an external table based on the output file and try to have that external table as your target to update rather than trying to update a file. At the back of the screen its all the same, ultimately your file will be updated. HOWEVER, I have not tried this, just throwing some ideas here.

Secondly, why not have a table as your target to capture your job execution statistics and populate that table only if the status is a failure, this will also enable you to query on that table and do further analysis on your job failures.

Alternatively, you can route all the failed source files to a separate folder, so that you not only know what has failed but you also have the source data to investigate further.

[Anonymous]

Hi,

   If my understanding about your requirement is correct, you want to log the success and error details for later verification. You can use either On SubJob Error or OnComponent Error component from the FTP flow to capture the error status. 

Generate one record using tRowgenerator with error status and load it into target DB. In above diagram, I am showing the skeleton diagram for your reference. If you need the same logic in multiple area, you can add the repeatable part to a joblet and call it with the help of context variables (to pass data dynamically).

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Thank you guys for your answers.

I'm not suppose to use a database ( for now at least ) to store my results and that's why I'm using a file.

@nthampi My requirement is to count the number of error in order to trigger a tsend mail. for example after 3 execution with an error for a file an email is sent.

[Anonymous]

@DjallelB 

If you have to just count the error, you can do it by storing it in a context variable and do a Run if check for that conext variable. If the value is not equal to zero, you can send the mail. You can even send the mail immediately from On Component Error by connecting to email component.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Yes thank you but the job runs every 15 min so the email has to be sent after a third or fourth execution in error.

[Anonymous]

Hi,

    I would prefer DB as there are always data recovery options available. But its not allowed, you can always go for file mode.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved