Six Stars

tMapRStreamsOutput in Spark Streaming Job

Hi,

 

It says here that it is possible to use tMapRStreamsOutput in an Apache Spark Streaming job.

But when I tried to create a job like this:

maprstreams.PNG

The job build fails with this error message:

org.talend.designer.runprocess.ProcessorException: Failed to generate code.
	at org.talend.designer.runprocess.java.JavaProcessor.generateCode(JavaProcessor.java:572)
	at org.talend.designer.runprocess.maven.MavenJavaProcessor.generateCode(MavenJavaProcessor.java:71)
	at org.talend.designer.runprocess.ProcessorUtilities.generateContextInfo(ProcessorUtilities.java:685)
	at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:511)
	at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:1413)
	at org.talend.designer.runprocess.RunProcessContext$1.run(RunProcessContext.java:576)
	at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:466)
	at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:374)
	at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:527)
	at org.eclipse.ui.internal.progress.ProgressMonitorJobsDialog.run(ProgressMonitorJobsDialog.java:284)
	at org.eclipse.ui.internal.progress.ProgressManager.run(ProgressManager.java:1190)
	at org.talend.designer.runprocess.RunProcessContext.exec(RunProcessContext.java:531)
	at org.talend.designer.runprocess.remote.RunRemoteProcessContext.exec(RunRemoteProcessContext.java:71)
	at org.talend.designer.runprocess.ui.ProcessComposite.exec(ProcessComposite.java:1450)
	at org.talend.designer.runprocess.ui.ProcessComposite.addInHistoryRunningList(ProcessComposite.java:1401)
	at org.talend.designer.runprocess.ui.ProcessComposite.execButtonPressed(ProcessComposite.java:935)
	at org.talend.designer.runprocess.ui.ProcessComposite.execRun(ProcessComposite.java:674)
	at org.talend.designer.runprocess.ui.ProcessComposite$1.widgetSelected(ProcessComposite.java:395)
	at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:248)
	at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
	at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4353)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1061)
	at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4172)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3761)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1151)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
	at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
	at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
	at org.talend.rcp.intro.Application.start(Application.java:255)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
Caused by: org.talend.designer.codegen.exception.CodeGeneratorException: org.eclipse.emf.codegen.jet.JETException: InvocationTargetException in null job
	at org.talend.designer.components.mrprovider.codegen.BigDataCodeGenerator.generateComponentCode(BigDataCodeGenerator.java:185)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentCodeWithHeader(AbstractSparkCodeGenerator.java:133)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:125)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkstrprovider.codegen.SparkStrCodeGen.generateProcessCode(SparkStrCodeGen.java:75)
	at org.talend.designer.runprocess.java.JavaProcessor.generateCode(JavaProcessor.java:569)
	... 44 more
Caused by: org.eclipse.emf.codegen.jet.JETException: InvocationTargetException
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:469)
	at org.talend.designer.codegen.config.TalendJetEmitter.generate(TalendJetEmitter.java:684)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:441)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:432)
	at org.talend.designer.codegen.proxy.JetProxy.generate(JetProxy.java:54)
	at org.talend.designer.components.mrprovider.codegen.BigDataCodeGenerator.generateComponentCode(BigDataCodeGenerator.java:181)
	... 68 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:461)
	... 73 more
Caused by: java.lang.NullPointerException
	at org.talend.designer.common.tsetkeystore.TSetKeystoreUtil.<init>(TSetKeystoreUtil.java:50)
	at org.talend.designer.spark.generator.storage.KafkaSparkStorage.<init>(KafkaSparkStorage.java:49)
	at org.talend.designer.codegen.translators.messaging.maprstreams.TMapRStreamsOutputSparkstreamingcodeJava.generate(TMapRStreamsOutputSparkstreamingcodeJava.java:147)
	... 78 more

But the job builds successfully if I remove the tMapRStreamsOutput component.

Or is it not possible to combine tMapRStreamsInput and tMapRStreamsOutput in a single job?

 

Thank you.

 

Tags (2)
3 REPLIES
Moderator

Re: tMapRStreamsOutput in Spark Streaming Job

Hello,

On which build version you got this issue?

Could you please take a look at this jira issue:https://jira.talendforge.org/browse/TBD-5521 to see if you have the same situation on your side?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: tMapRStreamsOutput in Spark Streaming Job

Hi Sabrina,

 

I ran into this issue on Talend Data Fabric 6.4.1.

Yes, the problem seems similar as my job fails to build and generate code too.

It says that the solution is to use Talend 6.4.2 or 6.5, whereas on the Download page only Talend 6.5 was available for download as a Milestone release type. Is the current solution only applicable for Milestone release type for now?

Moderator

Re: tMapRStreamsOutput in Spark Streaming Job

Hello,


veee wrote:

Hi Sabrina,

 

I ran into this issue on Talend Data Fabric 6.4.1.

Yes, the problem seems similar as my job fails to build and generate code too.

It says that the solution is to use Talend 6.4.2 or 6.5, whereas on the Download page only Talend 6.5 was available for download as a Milestone release type. Is the current solution only applicable for Milestone release type for now?


With your subscription solution Talend Data Fabric 6.4.1, could you please create a case on talend support portal? In this way, our colleagues from support team will schedule a webex and check your issue to see if it is possible to give a patch to you. https://login.talend.com/support-login.php.

Many thanks.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.