topic Re: How to log input and output files information in Talend Studio

How to log input and output files information

Anonymous — Sat, 16 Nov 2024 05:39:31 GMT

hi,

I have a job which converts Excel file to CSV file and i want to log all the files information mainly the count of input files,file path, filename, count and number of output files converted and reason if the job is not able to convert in case.

My job goes like this:

tfileList ----> tFileInputExcel ----- > tMap -----> tFileOutputDelimited(csv file created)

Please suggest a better way to handle logs information by adding any other components if needed.

Thanks!

Re: How to log input and output files information

Anonymous — Wed, 05 Jun 2019 16:27:48 GMT

I think you should check global variables

Re: How to log input and output files information

Anonymous — Tue, 11 Jun 2019 04:57:05 GMT

Hello @MNK_2019

Do your excel files have the same schema structure? Talend tFileInputExcel component doesn't support for dynamic schema feature.

Here is tFlowMeterCatcher component which catches the processing volumetric from the tFlowMeter component and passes them onto the output component.

For example:
tfileList ----> tFileInputExcel ----- > tMap ----->tFlowMeter--> tFileOutputDelimited(csv file created)
tFlowMeterCatcher---tLogRow
There is one column called "count" in the schema of tFlowMeterCatcher which counts the no. of records pass by the specify flow.
Let us know if it is OK with you.

Best regards

Sabrina

Re: How to log input and output files information

Anonymous — Tue, 11 Jun 2019 21:31:06 GMT

Thanks for the suggestion.

But my requirement is to log the details of files. i.e whether the files is converted successfully or not.

If the files is not converted successfully, then an error message is to be populated in my custom log table.

Any sort of logs information coming tfileoutputdelimited is fine for me.

Re: How to log input and output files information

Anonymous — Thu, 13 Jun 2019 10:32:23 GMT

Hello,

It seems you are looking for "RunIf" trigger to add a condition when your file is converted successfully or not.

Could you please elaborate your case with an example with expected output log message?

Best regards

Sabrina