How to log input and output files information

[Anonymous]

hi,

I have a job which converts Excel file to CSV file and i want to log all the files information mainly the count of input files,file path, filename, count and number of output files converted and reason if the job is not able to convert in case.

My job goes like this:

tfileList ---->  tFileInputExcel ----- > tMap ----->   tFileOutputDelimited(csv file created)

Please suggest a better way to handle logs information by adding any other components if needed.

Thanks!

[Anonymous]

I think you should check global variables

[Anonymous]

Hello @MNK_2019

Do your excel files have the same schema structure? Talend tFileInputExcel component doesn't support for dynamic schema feature.

Here is  tFlowMeterCatcher component which catches the processing volumetric from the tFlowMeter component and passes them onto the output component.

For example:
tfileList ----> tFileInputExcel ----- > tMap ----->tFlowMeter--> tFileOutputDelimited(csv file created)
tFlowMeterCatcher---tLogRow
There is one column called "count" in the schema of tFlowMeterCatcher which counts the no. of records pass by the specify flow.
Let us know if it is OK with you.

Best regards

Sabrina

[Anonymous]

Thanks for the suggestion.

But my requirement is to log the details of files. i.e whether the files is converted successfully or not.

If the files is not converted successfully, then an error message is to be populated in my custom log table.

Any sort of logs information coming tfileoutputdelimited is fine for me.

[Anonymous]

Hello,

It seems you are looking for "RunIf" trigger to add a condition when your file is converted successfully or not.

Could you please elaborate your case with an example with expected output log message?

Best regards

Sabrina