Talend 7.0 OS for BD and Hive: Cannot modify mapred.job.name at runtime

[Anonymous]

Hi Community,

I have, unfortunately, yet another problem after upgrading from Talend OS for Big Data 6.4 to 7.0. Jobs that include tHiveInput fail with the error message:

Caused by: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify mapred.job.name at runtime. It is not in lisat are allowed to be modified at runtime

Alright. I have done quite some research on this but everything points to changing the clusters permissions and appending "mapred.job.name" to the whitelisted parameters.

Unfortunately, this is not an option for us. 

Does anyone already have dealt with a similar problem and has a solution/workaround for this?

Thanks in advance!

Freddy

[Anonymous]

Hello,

Here exists a jira issue about:https://jira.talendforge.org/browse/TBD-622

Let us know if the provided solutions are OK with you.

Best regards

Sabrina

[Anonymous]

Hi Sabrina!

Thanks for the link. Unfortunately, the two given solutions did not help. The first one is to change the cluster-configuration, which, as mentioned, is not an option. 

The second solution is to uncheck all hadoop parameters in the connection view. They are all unchecked by default, except the "use datanode name". Since this led to other problems, I have already changed the settings to a built-in configuration and unchecked this option for every job a couple of weeks ago. Unfortunately, the error persists:

Starte Job BD_TSPJ0010_Vegdashboard_Hive_Oracle am 12:26 16/07/2018.

[statistics] connecting to socket on port 3539
[statistics] connected
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:262)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:248)
	at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:300)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
	at allgemeine_ladeprozesse.bd_tspj0010_vegdashboard_hive_oracle_0_1.BD_TSPJ0010_Vegdashboard_Hive_Oracle.tHiveInput_1Process(BD_TSPJ0010_Vegdashboard_Hive_Oracle.java:404)
	at allgemeine_ladeprozesse.bd_tspj0010_vegdashboard_hive_oracle_0_1.BD_TSPJ0010_Vegdashboard_Hive_Oracle.runJobInTOS(BD_TSPJ0010_Vegdashboard_Hive_Oracle.java:756)
	at allgemeine_ladeprozesse.bd_tspj0010_vegdashboard_hive_oracle_0_1.BD_TSPJ0010_Vegdashboard_Hive_Oracle.main(BD_TSPJ0010_Vegdashboard_Hive_Oracle.java:605)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify mapred.job.name at runtime. It is not in list of params that are allowed to be modified at runtime
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:323)
	at org.apache.hive.service.cli.operation.HiveCommandOperation.runInternal(HiveCommandOperation.java:108)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:264)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:470)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:457)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy29.executeStatementAsync(Unknown Source)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:313)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:509)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1317)
	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1302)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[statistics] disconnected

Job BD_TSPJ0010_Vegdashboard_Hive_Oracle endet am 12:26 16/07/2018. [exit code=0]

Any other ideas? If not, my only option seems to be to start using Talend 6.4 again, where the error did not occur.

Thanks in advance!

Freddy

[Anonymous]

It is possible to disable changing mapreduce.job.name and hive.query.name at runtime by Talend7 jobs.

Edit the file {talend_install_dir}/plugins/org.talend.designer.components.localprovider_7.1.1.20181026_1147/components/templates/Hive/SetQueryName.javajet

and comment out lines 6 and 11 like that:

// stmt_<%=cid %>.execute("set mapred.job.name=" + queryIdentifier_<%=cid %>);

// stmt_<%=cid %>.execute("set hive.query.name=" + queryIdentifier_<%=cid %>);

It solved this issue for me.