Hi, I would like to ask for advice on how to capture logs and statistics when running a job in Spark. I understand that the tLogCatcher component is not available in Spark jobs.
I would like to gather information such as start time, end time, and errors if any issues occur, and then store all the information into a log table without relying on any third-party software.
