Spark Batch Job has different logging level in console than in YARN JobHistory

Jaruro2810 — Tue, 03 Sep 2024 17:25:11 GMT

Hello,

I'm using version 7.3.1 with the 2020-09 patch applied. I'm using a remote connection to the to work on a Spark Batch job (spark ver 2.4.0) in a Cloudera Enterprise 6.3.1 enviroment.

This spark batch job has configured the default log level of WARN on console, since it's unchanged in advanced settings.

Yet, for some reason, on the JobHistory logs it shows up on TRACE level.

[INFO ] 13:08:32 org.apache.spark.deploy.yarn.ApplicationMaster- ApplicationAttemptId: appattempt_1724052715153_5149_000002
[DEBUG] 13:08:32 org.apache.spark.util.ShutdownHookManager- Adding shutdown hook
[TRACE] 13:08:32 org.apache.hadoop.security.SecurityUtil- Name lookup for <<private url>> took 0 ms.
[INFO ] 13:08:32 org.apache.spark.deploy.yarn.ApplicationMaster- Starting the user application in a separate Thread
[INFO ] 13:08:32 org.apache.spark.deploy.yarn.ApplicationMaster- Waiting for spark context initialization...

This is the ONLY job in our project that does this. The other spark batches correctly generate up to WARN level in JobHistory, including newly created ones.

I'd like to find the reason behind this. Any help is appreciated.

Thanks, Jaruro2810.

topic Spark Batch Job has different logging level in console than in YARN JobHistory in Talend Studio

Spark Batch Job has different logging level in console than in YARN JobHistory