tMapRStreamsOutput in Spark Streaming Job

Anonymous — Sat, 16 Nov 2024 09:09:15 GMT

Hi,

It says here that it is possible to use tMapRStreamsOutput in an Apache Spark Streaming job.

But when I tried to create a job like this:

The job build fails with this error message:

org.talend.designer.runprocess.ProcessorException: Failed to generate code.
	at org.talend.designer.runprocess.java.JavaProcessor.generateCode(JavaProcessor.java:572)
	at org.talend.designer.runprocess.maven.MavenJavaProcessor.generateCode(MavenJavaProcessor.java:71)
	at org.talend.designer.runprocess.ProcessorUtilities.generateContextInfo(ProcessorUtilities.java:685)
	at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:511)
	at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:1413)
	at org.talend.designer.runprocess.RunProcessContext$1.run(RunProcessContext.java:576)
	at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:466)
	at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:374)
	at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:527)
	at org.eclipse.ui.internal.progress.ProgressMonitorJobsDialog.run(ProgressMonitorJobsDialog.java:284)
	at org.eclipse.ui.internal.progress.ProgressManager.run(ProgressManager.java:1190)
	at org.talend.designer.runprocess.RunProcessContext.exec(RunProcessContext.java:531)
	at org.talend.designer.runprocess.remote.RunRemoteProcessContext.exec(RunRemoteProcessContext.java:71)
	at org.talend.designer.runprocess.ui.ProcessComposite.exec(ProcessComposite.java:1450)
	at org.talend.designer.runprocess.ui.ProcessComposite.addInHistoryRunningList(ProcessComposite.java:1401)
	at org.talend.designer.runprocess.ui.ProcessComposite.execButtonPressed(ProcessComposite.java:935)
	at org.talend.designer.runprocess.ui.ProcessComposite.execRun(ProcessComposite.java:674)
	at org.talend.designer.runprocess.ui.ProcessComposite$1.widgetSelected(ProcessComposite.java:395)
	at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:248)
	at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
	at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4353)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1061)
	at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4172)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3761)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1151)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
	at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
	at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
	at org.talend.rcp.intro.Application.start(Application.java:255)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
Caused by: org.talend.designer.codegen.exception.CodeGeneratorException: org.eclipse.emf.codegen.jet.JETException: InvocationTargetException in null job
	at org.talend.designer.components.mrprovider.codegen.BigDataCodeGenerator.generateComponentCode(BigDataCodeGenerator.java:185)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentCodeWithHeader(AbstractSparkCodeGenerator.java:133)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:125)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkstrprovider.codegen.SparkStrCodeGen.generateProcessCode(SparkStrCodeGen.java:75)
	at org.talend.designer.runprocess.java.JavaProcessor.generateCode(JavaProcessor.java:569)
	... 44 more
Caused by: org.eclipse.emf.codegen.jet.JETException: InvocationTargetException
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:469)
	at org.talend.designer.codegen.config.TalendJetEmitter.generate(TalendJetEmitter.java:684)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:441)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:432)
	at org.talend.designer.codegen.proxy.JetProxy.generate(JetProxy.java:54)
	at org.talend.designer.components.mrprovider.codegen.BigDataCodeGenerator.generateComponentCode(BigDataCodeGenerator.java:181)
	... 68 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:461)
	... 73 more
Caused by: java.lang.NullPointerException
	at org.talend.designer.common.tsetkeystore.TSetKeystoreUtil.<init>(TSetKeystoreUtil.java:50)
	at org.talend.designer.spark.generator.storage.KafkaSparkStorage.<init>(KafkaSparkStorage.java:49)
	at org.talend.designer.codegen.translators.messaging.maprstreams.TMapRStreamsOutputSparkstreamingcodeJava.generate(TMapRStreamsOutputSparkstreamingcodeJava.java:147)
	... 78 more

But the job builds successfully if I remove the tMapRStreamsOutput component.

Or is it not possible to combine tMapRStreamsInput and tMapRStreamsOutput in a single job?

Thank you.

Re: tMapRStreamsOutput in Spark Streaming Job

Anonymous — Tue, 24 Oct 2017 07:34:12 GMT

Hello,

On which build version you got this issue?

Could you please take a look at this jira issue:https://jira.talendforge.org/browse/TBD-5521 to see if you have the same situation on your side?

Best regards

Sabrina

Re: tMapRStreamsOutput in Spark Streaming Job

Anonymous — Wed, 25 Oct 2017 03:36:34 GMT

Hi Sabrina,

I ran into this issue on Talend Data Fabric 6.4.1.

Yes, the problem seems similar as my job fails to build and generate code too.

It says that the solution is to use Talend 6.4.2 or 6.5, whereas on the Download page only Talend 6.5 was available for download as a Milestone release type. Is the current solution only applicable for Milestone release type for now?

Re: tMapRStreamsOutput in Spark Streaming Job

Anonymous — Wed, 25 Oct 2017 09:11:58 GMT

Hello,

@veee wrote:

Hi Sabrina,

I ran into this issue on Talend Data Fabric 6.4.1.

Yes, the problem seems similar as my job fails to build and generate code too.

It says that the solution is to use Talend 6.4.2 or 6.5, whereas on the Download page only Talend 6.5 was available for download as a Milestone release type. Is the current solution only applicable for Milestone release type for now?

With your subscription solution Talend Data Fabric 6.4.1, could you please create a case on talend support portal? In this way, our colleagues from support team will schedule a webex and check your issue to see if it is possible to give a patch to you. https://login.talend.com/support-login.php.

Many thanks.

Best regards

Sabrina

topic Re: tMapRStreamsOutput in Spark Streaming Job in Talend Studio

tMapRStreamsOutput in Spark Streaming Job

Re: tMapRStreamsOutput in Spark Streaming Job

Re: tMapRStreamsOutput in Spark Streaming Job

Re: tMapRStreamsOutput in Spark Streaming Job