Six Stars

Spark Streaming Job Not Working

Hi,

Trying to replicate spark streaming job available HERE

Following are details from log file (Detail Log File & Spark Configuration attached)

[ERROR]: org.apache.spark.streaming.scheduler.ReceiverTracker - Deregistered receiver for stream 0: Stopped by driver
[ERROR]: org.apache.spark.scheduler.cluster.YarnScheduler - Lost executor 1 on 172.18.0.8: Container marked as failed: container_1516284572350_0001_01_000002 on host: talend-cdh580.weave.local. Exit status: 50. Diagnostics: Exception from container-launch.

 

Can someone help me understand why the job is throwing this error. Thank you!!!

1 REPLY
Six Stars

Re: Spark Streaming Job Not Working

Progress - Removed the spark web UI property and increased the memory. Now getting following error (Detail log File Attached)

 

[Stage 0:====================================================> (46 + 2) / 50]
[Stage 0:=====================================================> (47 + 2) / 50]
[Stage 0:======================================================> (48 + 2) / 50]
[Stage 0:=======================================================> (49 + 1) / 50]
[Stage 1Smiley Embarassed (0 + 0) / 20]
[Stage 1Smiley Embarassed (0 + 2) / 20]
[Stage 1:==> (1 + 2) / 20]
[Stage 1:========> (3 + 2) / 20]
[Stage 1:=================> (6 + 2) / 20]
[Stage 1:====================> (7 + 2) / 20]
[Stage 1:=======================> (8 + 3) / 20]
[Stage 1:=======================> (8 + 4) / 20]
[Stage 1:===============================> (11 + 2) / 20]
[Stage 1:==================================> (12 + 2) / 20]
[Stage 1:=====================================> (13 + 3) / 20]
[Stage 1:=======================================> (14 + 3) / 20]
[Stage 1:================================================> (17 + 2) / 20]
[Stage 1:===================================================> (18 + 2) / 20]
[Stage 1:======================================================> (19 + 1) / 20]

[ERROR]: org.apache.spark.streaming.scheduler.ReceiverTracker - Deregistered receiver for stream 0: Stopped by driver
Exception in thread "streaming-job-executor-0" java.lang.Error: java.lang.InterruptedException
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1148)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)