Cannot Get Talend Spark Job Execution Logs in the Spark History Server

Talend Version          6.2.1

Summary

When executing a Talend Spark Job, the Job is not displayed in the Spark History Server console.

 

This article shows how to configure a Talend Spark Job so that logs related to the Job execution can be accessed using the Spark History Server console.

Additional Versions 6.3.1
Product Talend Big Data
Component Spark
Problem Description When executing a Talend Spark Job, the Job is not displayed in the Spark History Server console.
Problem root cause In Talend 6.2.1 and above, configuration properties have been provided to enable Spark event logging for a Talend Spark Job. If this Job configuration property is not enabled, the Job will not appear in the Spark History Server console after its execution.
Solution or Workaround

Use an option in Talend Studio to enable Spark History logs related to Talend Spark Job execution:

  1. Select the Spark Job to run.
  2. Click Run Job > Spark Configuration.
  3. In the Spark history section, select Enable spark event logging.
  4. Enter the Spark event logs directory (for example, hdfs://namenode:8020/user/spark/applicationHistory).
  5. Enter the Spark history server address (for example, sparkHistoryServer:18080).

    Here is an image showing this Spark configuration :

    sparkhistory.png

     

JIRA ticket number TBD-3608
Version history
Revision #:
21 of 21
Last update:
‎11-14-2017 07:17 AM
Updated by:
 
Labels (3)
Contributors