IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

Five Stars

IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

Hi,

 

I am using the

Talend Big Data and Machine Learning Sandbox 

This step

IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

fails with the following error

org.talend.designer.runprocess.ProcessorException: Job compile errors
At least job "Step_02_Train_PredictiveMaintenance" has a compile errors, please fix and export again.
Error Line: 1235
Detail Message: org.apache.spark.ml.linalg.Vector cannot be resolved to a type
There may be some other errors caused by JVM compatibility. Make sure your JVM setup is similar to the studio.
at org.talend.designer.runprocess.JobErrorsChecker.checkLastGenerationHasCompilationError(JobErrorsChecker.java:338)
at org.talend.designer.runprocess.DefaultRunProcessService.checkLastGenerationHasCompilationError(DefaultRunProcessService.java:464)
at org.talend.designer.runprocess.RunProcessService.checkLastGenerationHasCompilationError(RunProcessService.java:316)
at org.talend.designer.runprocess.ProcessorUtilities.generateBuildInfo(ProcessorUtilities.java:812)
at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:586)
at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:1736)
at org.talend.designer.runprocess.RunProcessContext$1.run(RunProcessContext.java:582)
at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:466)
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:374)
at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:527)
at org.eclipse.ui.internal.progress.ProgressMonitorJobsDialog.run(ProgressMonitorJobsDialog.java:284)
at org.eclipse.ui.internal.progress.ProgressManager.run(ProgressManager.java:1190)
at org.talend.designer.runprocess.RunProcessContext.exec(RunProcessContext.java:534)
at org.talend.designer.runprocess.remote.RunRemoteProcessContext.exec(RunRemoteProcessContext.java:84)
at org.talend.designer.runprocess.ui.ProcessComposite.exec(ProcessComposite.java:1401)
at org.talend.designer.runprocess.ui.ProcessComposite.addInHistoryRunningList(ProcessComposite.java:1352)
at org.talend.designer.runprocess.ui.ProcessComposite.execButtonPressed(ProcessComposite.java:930)
at org.talend.designer.runprocess.ui.ProcessComposite.execRun(ProcessComposite.java:680)
at org.talend.designer.runprocess.ui.ProcessComposite$1.widgetSelected(ProcessComposite.java:396)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:248)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4454)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1388)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3799)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3409)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1151)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
at org.talend.rcp.intro.Application.start(Application.java:265)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
at org.eclipse.equinox.launcher.Main.main(Main.java:1438)

 

Please provide resolution


Accepted Solutions
Employee

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

Hi @aslame,

 

Great it worked for you! Yes these warnings are common in all Hadoop and Spark environments and are not impacting the job itself.

 

Please, set your question as resolved if all is working for you now,

 

Don't hesitate to ask a new question if you encounter any other issue,

 

Thank you,

 

Thibaut


All Replies
Employee

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

Hello @aslame,

 

Thank you for testing our Big Data & Machine Learning Sandbox.

 

We are sorry this issue happenned. There is a dependency issue that broke our job. Until our team fix the issue you can use the following workaround:

 

In the following Big Data jobs:

  • Step_02_Train_PredictiveMaintenance
  • Step_02bis_Test_PredictiveMaintenance
  • Step_03_PredictMachinesMaintenance

You need to add a tLibraryLoad component with spark-mllib-local_2.11-2.2.0.jar. Please find attached a screenshot of the workaround.

 

sandbox_workaround.png

Five Stars

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

Hi,

 

I added tLibraryLoad component with spark-mllib-local_2.11-2.2.0.cloudera1.jar

I do not have spark-mllib-local_2.11-2.2.0.cloudera1.jar. Also tried spark-mllib_2.11-2.2.0.cloudera1.jar

  but it is still failing

 

Highlighted
Employee

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

Hello,

 

You should not use spark-mllib-local_2.11-2.2.0.cloudera1.jar but specifically choose spark-mllib-local_2.11-2.2.0.jar. Can you try with this one instead.

 

Thank you,

 

Thibaut

 

Five Stars

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

I do not have spark-mllib-local_2.11-2.2.0.jar.
Where do I get spark-mllib-local_2.11-2.2.0.jar from and how do I load it.
I am new to Talend, so please provide more detailed explanation.
Thanks, Aslam
Five Stars

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

spark-mllib-local_2.11-2.2.0.jar does not show up in the list.
Employee

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

If you don't have it in the dropdown list, you can download it here. You just have to use tLibraryLoad as well and select Artifact Repository (local/m2), then in Install a new module select the jar file you downloaded.

 

Sorry for the troubles, let me know if it works this time,

 

Thibaut

Five Stars

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

Hi Thibaut,

 

It sort of works, but at the beginning I get the following warnings and error java errors.

Is this normal?

 

Starting job Step_02_Train_PredictiveMaintenance at 06:54 10/11/2018.

[statistics] connecting to socket on port 3651
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
[WARN ]: org.apache.spark.util.Utils - Your hostname, talend resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
[WARN ]: org.apache.spark.util.Utils - Set SPARK_LOCAL_IP if you need to bind to another address
[WARN ]: org.apache.spark.util.Utils - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
[WARN ]: org.apache.hadoop.hdfs.DFSClient - Caught exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:952)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeInternal(DFSOutputStream.java:920)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:916)

Employee

Re: IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

Hi @aslame,

 

Great it worked for you! Yes these warnings are common in all Hadoop and Spark environments and are not impacting the job itself.

 

Please, set your question as resolved if all is working for you now,

 

Don't hesitate to ask a new question if you encounter any other issue,

 

Thank you,

 

Thibaut

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now