IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

[aslame]

Hi,

I am using the

Talend Big Data and Machine Learning Sandbox 

This step

IoTPredictiveMaintenance - Step_02_Train_PredictiveMaintenance

fails with the following error

org.talend.designer.runprocess.ProcessorException: Job compile errors
At least job "Step_02_Train_PredictiveMaintenance" has a compile errors, please fix and export again.
Error Line: 1235
Detail Message: org.apache.spark.ml.linalg.Vector cannot be resolved to a type
There may be some other errors caused by JVM compatibility. Make sure your JVM setup is similar to the studio.
at org.talend.designer.runprocess.JobErrorsChecker.checkLastGenerationHasCompilationError(JobErrorsChecker.java:338)
at org.talend.designer.runprocess.DefaultRunProcessService.checkLastGenerationHasCompilationError(DefaultRunProcessService.java:464)
at org.talend.designer.runprocess.RunProcessService.checkLastGenerationHasCompilationError(RunProcessService.java:316)
at org.talend.designer.runprocess.ProcessorUtilities.generateBuildInfo(ProcessorUtilities.java:812)
at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:586)
at org.talend.designer.runprocess.ProcessorUtilities.generateCode(ProcessorUtilities.java:1736)
at org.talend.designer.runprocess.RunProcessContext$1.run(RunProcessContext.java:582)
at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:466)
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:374)
at org.eclipse.jface.dialogs.ProgressMonitorDialog.run(ProgressMonitorDialog.java:527)
at org.eclipse.ui.internal.progress.ProgressMonitorJobsDialog.run(ProgressMonitorJobsDialog.java:284)
at org.eclipse.ui.internal.progress.ProgressManager.run(ProgressManager.java:1190)
at org.talend.designer.runprocess.RunProcessContext.exec(RunProcessContext.java:534)
at org.talend.designer.runprocess.remote.RunRemoteProcessContext.exec(RunRemoteProcessContext.java:84)
at org.talend.designer.runprocess.ui.ProcessComposite.exec(ProcessComposite.java:1401)
at org.talend.designer.runprocess.ui.ProcessComposite.addInHistoryRunningList(ProcessComposite.java:1352)
at org.talend.designer.runprocess.ui.ProcessComposite.execButtonPressed(ProcessComposite.java:930)
at org.talend.designer.runprocess.ui.ProcessComposite.execRun(ProcessComposite.java:680)
at org.talend.designer.runprocess.ui.ProcessComposite$1.widgetSelected(ProcessComposite.java:396)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:248)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4454)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1388)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3799)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3409)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1151)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
at org.talend.rcp.intro.Application.start(Application.java:265)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
at org.eclipse.equinox.launcher.Main.main(Main.java:1438)

Please provide resolution

[tgourdel · Accepted solution]

Hi @aslame,

Great it worked for you! Yes these warnings are common in all Hadoop and Spark environments and are not impacting the job itself.

Please, set your question as resolved if all is working for you now,

Don't hesitate to ask a new question if you encounter any other issue,

Thank you,

Thibaut

[tgourdel]

Hello @aslame,

Thank you for testing our Big Data & Machine Learning Sandbox.

We are sorry this issue happenned. There is a dependency issue that broke our job. Until our team fix the issue you can use the following workaround:

In the following Big Data jobs:

• Step_02_Train_PredictiveMaintenance
• Step_02bis_Test_PredictiveMaintenance
• Step_03_PredictMachinesMaintenance

You need to add a tLibraryLoad component with spark-mllib-local_2.11-2.2.0.jar. Please find attached a screenshot of the workaround.

[aslame]

Hi,

I added tLibraryLoad component with spark-mllib-local_2.11-2.2.0.cloudera1.jar

I do not have spark-mllib-local_2.11-2.2.0.cloudera1.jar. Also tried spark-mllib_2.11-2.2.0.cloudera1.jar

  but it is still failing

[tgourdel]

Hello,

You should not use spark-mllib-local_2.11-2.2.0.cloudera1.jar but specifically choose spark-mllib-local_2.11-2.2.0.jar. Can you try with this one instead.

Thank you,

Thibaut

[aslame]

I do not have spark-mllib-local_2.11-2.2.0.jar.
Where do I get spark-mllib-local_2.11-2.2.0.jar from and how do I load it.
I am new to Talend, so please provide more detailed explanation.
Thanks, Aslam

[aslame]

spark-mllib-local_2.11-2.2.0.jar does not show up in the list.

[tgourdel]

If you don't have it in the dropdown list, you can download it here. You just have to use tLibraryLoad as well and select Artifact Repository (local/m2), then in Install a new module select the jar file you downloaded.

Sorry for the troubles, let me know if it works this time,

Thibaut

[aslame]

Hi Thibaut,

It sort of works, but at the beginning I get the following warnings and error java errors.

Is this normal?

Starting job Step_02_Train_PredictiveMaintenance at 06:54 10/11/2018.

[statistics] connecting to socket on port 3651
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
[WARN ]: org.apache.spark.util.Utils - Your hostname, talend resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
[WARN ]: org.apache.spark.util.Utils - Set SPARK_LOCAL_IP if you need to bind to another address
[WARN ]: org.apache.spark.util.Utils - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
[WARN ]: org.apache.hadoop.hdfs.DFSClient - Caught exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:952)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeInternal(DFSOutputStream.java:920)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:916)

[tgourdel · Accepted solution]

Hi @aslame,

Great it worked for you! Yes these warnings are common in all Hadoop and Spark environments and are not impacting the job itself.

Please, set your question as resolved if all is working for you now,

Don't hesitate to ask a new question if you encounter any other issue,

Thank you,

Thibaut