Merging logs from tStatCatcher and tLogCatcher to a PostgreSQL table

Anonymous — Fri, 03 Feb 2017 14:16:44 GMT

Hi,
I have faced another issue which I don't quite understand. My usecase is that I am trying to create a table that will have all the logs of the jobs, even if they succeeded or failed. For this case, I join tStatCatcher and tLogCatcher with tMap and import the data in a PostgreSQL db.
My workflow looks like this

And the join in the tMap is like this:

So let's say that my job fails and I get this output in Talend:






p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Courier; color: #0433ff}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Courier; min-height: 12.0px}
p.p3 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Courier}
p.p4 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Courier; color: #ff2600}
span.Apple-tab-span {white-space:pre}


Starte Job Master_ETL_Job am 14:21 03/02/2017.


 connecting to socket on port 3607

 connected

2017-02-03 14:21:05|GV6iwG|GV6iwG|GV6iwG|7097|NASIA|Master_ETL_Job|_52dYEJUvEeaqS8phzVFskQ|0.1|Default||begin||

Exception in component tFileInputDelimited_1

java.io.FileNotFoundException: /Users/nasiantalla/Documents/keychain.csv (No such file or directory)

	at java.io.FileInputStream.open0(Native Method)

	at java.io.FileInputStream.open(FileInputStream.java:195)

	at java.io.FileInputStream.<init>(FileInputStream.java:138)

	at java.io.FileInputStream.<init>(FileInputStream.java:93)

	at org.talend.fileprocess.TOSDelimitedReader.<init>(TOSDelimitedReader.java:88)

	at org.talend.fileprocess.FileInputDelimited.<init>(FileInputDelimited.java:164)

	at nasia.master_etl_job_0_1.Master_ETL_Job.tFileInputDelimited_1Process(Master_ETL_Job.java:796)

	at nasia.master_etl_job_0_1.Master_ETL_Job.runJobInTOS(Master_ETL_Job.java:6119)

	at nasia.master_etl_job_0_1.Master_ETL_Job.main(Master_ETL_Job.java:5925)

2017-02-03 14:21:05|GV6iwG|GV6iwG|GV6iwG|NASIA|Master_ETL_Job|Default|6|Java Exception|tFileInputDelimited_1|java.io.FileNotFoundException:/Users/nasiantalla/Documents/keychain.csv (No such file or directory)|1

2017-02-03 14:21:05|GV6iwG|GV6iwG|GV6iwG|7097|NASIA|Master_ETL_Job|_52dYEJUvEeaqS8phzVFskQ|0.1|Default||end|failure|630

 disconnected

Job Master_ETL_Job endet am 14:21 03/02/2017.

However the output I get in my table in the database doesnt include the tLogCatcher data:

Do you see something that I miss here maybe?
Thanks a lot!!

Re: Merging logs from tStatCatcher and tLogCatcher to a PostgreSQL table

Anonymous — Fri, 03 Feb 2017 15:14:14 GMT

You shouldn't be joining your log and stat data like this. A tMap is driven by one source. In this case it is your stat data. If at the time the subjob starts for the stat data there is no log data, nothing will be sent to the db. You must have your data going to different tables and have it completely unlinked within Talend unless you are wanting to do something funky. The join between the two data sets is there to be linked when you want analyse the data after it has landed in the db.
I have written a tutorial on how you can use the AMC (Activity Monitoring Console) which deals handles all of this for you. You can find it here https://www.rilhia.com/tutorials/talend-activity-monitoring-console-amc

topic Re: Merging logs from tStatCatcher and tLogCatcher to a PostgreSQL table in Talend Studio

Merging logs from tStatCatcher and tLogCatcher to a PostgreSQL table

Re: Merging logs from tStatCatcher and tLogCatcher to a PostgreSQL table