topic Re: tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log in Talend Studio

tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log

asadri1 — Sat, 16 Nov 2024 04:44:47 GMT

When executing the Sqoop job using tsqoopimport I see the following in the logs. How can I avoid it? Is there a setup missing?

...

[INFO ]: org.apache.hadoop.mapred.LocalJobRunner -
[INFO ]: org.apache.hadoop.mapred.LocalJobRunner -
[INFO ]: org.apache.hadoop.mapred.LocalJobRunner -
[INFO ]: org.apache.hadoop.mapred.LocalJobRunner -

...

Re: tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log

Anonymous — Mon, 30 Sep 2019 07:53:55 GMT

Hello,

In Studio, you have the option to disable the log4j properties and that disables the logging for the Job design. In this case, Studio Project properties are set to disable, so the intended logs are not shown.

Best regards

Sabrina

Re: tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log

Anonymous — Mon, 30 Sep 2019 16:27:44 GMT

Hi,

You can configure the level of log4j logging by selecting the appropriate level as shown below.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log

asadri1 — Mon, 30 Sep 2019 16:39:05 GMT

Let me explain more: The log file indicates that the Sqoop is running on the local mode, not cluster mode. What setup do I need to do for the tsqoopimport to use cluster mode?

Re: tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log

Anonymous — Mon, 30 Sep 2019 16:46:15 GMT

Hi,

Could you please check the Spark Configuration in Run Tab of the job? You can remove the local mode option and then point the cluster parameters to run it either in Yarn Cluster or Yarn client mode.

I would recommend you to check the Big Data Basics Course to get familiar with the main concepts.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log

asadri1 — Mon, 30 Sep 2019 16:50:28 GMT

Sqoop uses MapReduce not Spark.

Re: tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log

asadri1 — Mon, 30 Sep 2019 19:01:02 GMT

Sqoop running on Local mode vs cluster mode is not the same thing as switching "Taget Exec" for a Talend job from TOS.

Presently Sqoop cannot use Spark engine:

https://issues.apache.org/jira/browse/SQOOP-1532

Our jobs using tsqoopimport somehow runs on Sqoop "local" mode rather than cluster. What needs to be setup to run tsqoopimport in cluster mode?