tStatCatcher, tLogCatcher, and tMSSqlClose

[Anonymous]

It appears that the tStatCatcher and tLogCatcher are actually run only at the end of a job -- which sort of makes sense given what they collect.  Can anyone confirm?  The problem this appears to create is, when using tMSSqlClose component to end the connection at post-job, it is getting fired before one or the other of the tStat or tLog are completed.

I would have expected that everything was completed before the tPostJob is fired.

Job design

Error because connection was closed before tStatCatcher was able to write to DB

If I remove the tMSSqlClose, it runs without errors -- but then the connection is not closed, right?

No error when tMSSqlClose removed

[Jesperrekuh]

Confirmed! And annoying I know.
If your crash is severe (a dirty exit) , everything is gone.
Try using log4j (in community version not supported but possible, in payed it is) and have a process which actually monitors the tail of the log-file (outside this talend job).

One thing some components just write the error to console/stdout which will never end up in your tlogcatcher ... Logging and provenance is horrible in Talend.

[Anonymous]

We've done something very similar, but rather than writing to a database, we write to a Kafka topic. We then have a consumer that reads from the Kafka topic and pushes the log messages to the ELK stack that comes with the TAC, but you could easily write a consumer that writes them to the database of your choice.  Better still, once you have the log messages on the Kafka topic, you could write them to both an ELK setup and database at the same time.