tMapRStreamsOutput in Spark Streaming Job

Highlighted
Six Stars

tMapRStreamsOutput in Spark Streaming Job

Hi,

 

It says here that it is possible to use tMapRStreamsOutput in an Apache Spark Streaming job.

But when I tried to create a job like this:

maprstreams.PNG

The job build fails with this error message:

org.talend.designer.runprocess.ProcessorException: Failed to generate code.
	at org.talend.designer.runprocess.java.JavaProcessor.generateCode(JavaProcessor.java:572)
	at org.talend.designer.runprocess.maven.MavenJavaProcessor.generateCode(MavenJavaProcessor.java:71)
	at org.talend.designer.runprocess.ProcessorUtilities.generateContextInfo(ProcessorUtilities.java:685)
	at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:511)
	at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:1413)
	at org.talend.designer.runprocess.RunProcessContext$1.run(RunProcessContext.java:576)
	at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:466)
	at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:374)
	at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:527)
	at org.eclipse.ui.internal.progress.ProgressMonitorJobsDialog.run(ProgressMonitorJobsDialog.java:284)
	at org.eclipse.ui.internal.progress.ProgressManager.run(ProgressManager.java:1190)
	at org.talend.designer.runprocess.RunProcessContext.exec(RunProcessContext.java:531)
	at org.talend.designer.runprocess.remote.RunRemoteProcessContext.exec(RunRemoteProcessContext.java:71)
	at org.talend.designer.runprocess.ui.ProcessComposite.exec(ProcessComposite.java:1450)
	at org.talend.designer.runprocess.ui.ProcessComposite.addInHistoryRunningList(ProcessComposite.java:1401)
	at org.talend.designer.runprocess.ui.ProcessComposite.execButtonPressed(ProcessComposite.java:935)
	at org.talend.designer.runprocess.ui.ProcessComposite.execRun(ProcessComposite.java:674)
	at org.talend.designer.runprocess.ui.ProcessComposite$1.widgetSelected(ProcessComposite.java:395)
	at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:248)
	at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
	at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4353)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1061)
	at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4172)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3761)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1151)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
	at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
	at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
	at org.talend.rcp.intro.Application.start(Application.java:255)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
Caused by: org.talend.designer.codegen.exception.CodeGeneratorException: org.eclipse.emf.codegen.jet.JETException: InvocationTargetException in null job
	at org.talend.designer.components.mrprovider.codegen.BigDataCodeGenerator.generateComponentCode(BigDataCodeGenerator.java:185)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentCodeWithHeader(AbstractSparkCodeGenerator.java:133)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:125)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generatesTreeCode(AbstractSparkCodeGenerator.java:160)
	at org.talend.designer.components.sparkprovider.codegen.AbstractSparkCodeGenerator.generateComponentsCode(AbstractSparkCodeGenerator.java:126)
	at org.talend.designer.components.sparkstrprovider.codegen.SparkStrCodeGen.generateProcessCode(SparkStrCodeGen.java:75)
	at org.talend.designer.runprocess.java.JavaProcessor.generateCode(JavaProcessor.java:569)
	... 44 more
Caused by: org.eclipse.emf.codegen.jet.JETException: InvocationTargetException
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:469)
	at org.talend.designer.codegen.config.TalendJetEmitter.generate(TalendJetEmitter.java:684)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:441)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:432)
	at org.talend.designer.codegen.proxy.JetProxy.generate(JetProxy.java:54)
	at org.talend.designer.components.mrprovider.codegen.BigDataCodeGenerator.generateComponentCode(BigDataCodeGenerator.java:181)
	... 68 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.eclipse.emf.codegen.jet.JETEmitter.generate(JETEmitter.java:461)
	... 73 more
Caused by: java.lang.NullPointerException
	at org.talend.designer.common.tsetkeystore.TSetKeystoreUtil.<init>(TSetKeystoreUtil.java:50)
	at org.talend.designer.spark.generator.storage.KafkaSparkStorage.<init>(KafkaSparkStorage.java:49)
	at org.talend.designer.codegen.translators.messaging.maprstreams.TMapRStreamsOutputSparkstreamingcodeJava.generate(TMapRStreamsOutputSparkstreamingcodeJava.java:147)
	... 78 more

But the job builds successfully if I remove the tMapRStreamsOutput component.

Or is it not possible to combine tMapRStreamsInput and tMapRStreamsOutput in a single job?

 

Thank you.

 

Moderator

Re: tMapRStreamsOutput in Spark Streaming Job

Hello,

On which build version you got this issue?

Could you please take a look at this jira issue:https://jira.talendforge.org/browse/TBD-5521 to see if you have the same situation on your side?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: tMapRStreamsOutput in Spark Streaming Job

Hi Sabrina,

 

I ran into this issue on Talend Data Fabric 6.4.1.

Yes, the problem seems similar as my job fails to build and generate code too.

It says that the solution is to use Talend 6.4.2 or 6.5, whereas on the Download page only Talend 6.5 was available for download as a Milestone release type. Is the current solution only applicable for Milestone release type for now?

Moderator

Re: tMapRStreamsOutput in Spark Streaming Job

Hello,


@veee wrote:

Hi Sabrina,

 

I ran into this issue on Talend Data Fabric 6.4.1.

Yes, the problem seems similar as my job fails to build and generate code too.

It says that the solution is to use Talend 6.4.2 or 6.5, whereas on the Download page only Talend 6.5 was available for download as a Milestone release type. Is the current solution only applicable for Milestone release type for now?


With your subscription solution Talend Data Fabric 6.4.1, could you please create a case on talend support portal? In this way, our colleagues from support team will schedule a webex and check your issue to see if it is possible to give a patch to you. https://login.talend.com/support-login.php.

Many thanks.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now