I'm trying to design a simple Spark streaming job on Sandbox - generating rows using rowgenerator and loading it to HDFS. Attached is a spark configuration. I executed the job but getting following error -
Exception in thread "streaming-job-executor-0" java.lang.Error: java.lang.InterruptedException
[ERROR]: org.apache.spark.scheduler.cluster.YarnScheduler - Lost executor 1 on 172.18.0.7: Container marked as failed: container_1497799969116_0004_01_000002 on host: talend-cdh580.weave.local. Exit status: 50. Diagnostics: Exception from container-launch.
Container id: container_1497799969116_0004_01_000002
Exit code: 50
Detail log file is attached
Can someone please help me this error?
Hello! Thanks for the detailed information about the problem -- can you show us more information about the Spark tuning settings that you've chosen? I'm thinking that there might not be enough executors to run the receiver stream (I noted that there was just one core per executor, but not how many executors).
Try unclicking your custom spark tuning to see if the default settings are OK -- your job is pretty much how I test my setup, so it should be OK!
Best regards, Ryan