topic Re: Fetch and store result from log.info() into mysql database using talend studio in Talend Studio

Fetch and store result from log.info() into mysql database using talend studio

WSyahirah21 — Fri, 15 Nov 2024 23:02:43 GMT

Currently, my spark job managed to output result from the log.info() function. What I was trying to do now is to fetch and store the output into a proper mysql database using talend component.

Result from spark:

INFO:spark2:Source rows = 2692687

INFO:jspark2:Destination rows = 2692687

My idea is to use these workflow to achieve the task objective:

tSystem (submit spark job) >> tFileInputDelimited (fetch the printing result from spark) >> <some mysql component to store result to msql>

Is it possible to do this way?

Re: Fetch and store result from log.info() into mysql database using talend studio

Anonymous — Fri, 08 Apr 2022 08:54:13 GMT

Hello,

You can read and parse the logs produced by spark and use a tFileInputDelimited -> tFilterRow -> tDBOutput(MySQL)

Another option would be to use the outputLine of tSystem if the job you call produces console logs.

Log4j messages themselves could be forwarded to a database. One would have to use JDBCAppender for log4j.

In case of Log4J2 the config I used (for a POC!) was:

<DriverManager connectionString="jdbc:postgresql://localhost:5432/postgres"

driverClassName="org.postgresql.Driver" username="postgres" password=":)" />

</MapFilter>

</Filters>

</JDBC>

This config is not for production use as it creates a lot of new connections.

Re: Fetch and store result from log.info() into mysql database using talend studio

WSyahirah21 — Sat, 09 Apr 2022 08:43:05 GMT

Hi,

currently I found other alternative to fetch the string output using:

tSystem_1 (print the output) -> tJava (get the output and bring to next component)

tSystem_1 output:

('source count:', '1000')

('destination count:', '100')

tJava code:

String output=((String)globalMap.get("tSystem_1_OUTPUT"));

System.out.println("Printing the error code 1 : "+StringUtils.substringBetween(output,"source count:", "destination count:"));

tJava output:

Printing the error code 1 : ', '1000')

However, tJava takes the result exactly between string " source count:' " and " destination count: ", so that included the bracket and all. The expected result is only to get the value 1000.

How do I apply this java code to fetch the correct output?