tsqoopimport - org.apache.hadoop.mapred.LocalJobRunner in the log

[asadri1]

When executing the Sqoop job using tsqoopimport I see the following in the logs.  How can I avoid it?  Is there a setup missing?

...

[INFO ]: org.apache.hadoop.mapred.LocalJobRunner -
[INFO ]: org.apache.hadoop.mapred.LocalJobRunner -
[INFO ]: org.apache.hadoop.mapred.LocalJobRunner -
[INFO ]: org.apache.hadoop.mapred.LocalJobRunner -

...

[Anonymous]

Hello,

In Studio, you have the option to disable the log4j properties and that disables the logging for the Job design. In this case, Studio Project properties are set to disable, so the intended logs are not shown.

Best regards

Sabrina

[Anonymous]

Hi,

    You can configure the level of log4j logging by selecting the appropriate level as shown below.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[asadri1]

Let me explain more: The log file indicates that the Sqoop is running on the local mode, not cluster mode. What setup do I need to do for the tsqoopimport to use cluster mode?

[Anonymous]

Hi,

   Could you please check the Spark Configuration in Run Tab of the job? You can remove the local mode option and then point the cluster parameters to run it either in Yarn Cluster or Yarn client mode.

   I would recommend you to check the Big Data Basics Course to get familiar with the main concepts.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[asadri1]

Sqoop uses MapReduce not Spark.

[asadri1]

Sqoop running on Local mode vs cluster mode is not the same thing as switching "Taget Exec" for a  Talend job from TOS. 

Presently Sqoop cannot use Spark engine:

https://issues.apache.org/jira/browse/SQOOP-1532

Our jobs using tsqoopimport somehow runs on Sqoop "local" mode rather than cluster. What needs to be setup to run tsqoopimport in cluster mode?