A lot of Messages shown in the Big Data Platform studio

Highlighted
Six Stars

A lot of Messages shown in the Big Data Platform studio

Hi all, 

 

I'm trying to process a big data with spark batch like below.

During the job running, I saw a lot of messages in the console.

 

 

spark image.PNG

 

What are these messages?

What do these messages means?

 

I'm not sure if this job is working well or not.

 

Could someone explain this situation??

 

Thank you in advance.

Employee

Re: A lot of Messages shown in the Big Data Platform studio

Hi,

 

    It looks like you are printing the data somewhere in the job. 

 

     Could you please check whether you are using tLogrow component or any other component or code (intjava or tJavarow etc) where you have given printing instructions.

 

     Also please verify whether there is any tflowmeter component added to your job. 

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: A lot of Messages shown in the Big Data Platform studio

Hi,

 

Thank you for your reply.

 

I'm not using any tlogrow or any other components which prints log in my job.

 

I see this logs in console when this error message appears..

Unable to connect to localhost on the port 3807 

 

[statistics] connecting to socket on port 3807
[statistics] connection refused
[INFO ]: org.apache.spark.SparkContext - Added file /home/hadoop/Talend-JobServer-20181026_1147-V7.1.1/TalendJobServersFiles/repository/DWH_Lyr1_Intg_BDB_gaw_t_so_detail_header_merge_20190220_184110_UTrPh/Lyr1_Intg_BDB_gaw_t_so_detail_header_merge/dwh/lyr1_intg_bdb_gaw_t_so_detail_header_merge_0_1/contexts/Default.properties at spark://ip-10-118-121-22.ap-northeast-1.compute.internal:44933/files/Default.properties with timestamp 1550655728404
[INFO ]: org.apache.spark.util.Utils - Copying /home/hadoop/Talend-JobServer-20181026_1147-V7.1.1/TalendJobServersFiles/repository/DWH_Lyr1_Intg_BDB_gaw_t_so_detail_header_merge_20190220_184110_UTrPh/Lyr1_Intg_BDB_gaw_t_so_detail_header_merge/dwh/lyr1_intg_bdb_gaw_t_so_detail_header_merge_0_1/contexts/Default.properties to /mnt/tmp/spark-06a1eaba-9e28-4ef0-a32d-b6f5cd008a07/userFiles-7d80b3b4-4f37-49f1-bd3c-982bcb37b2e8/Default.properties
[INFO ]: dwh.lyr1_intg_bdb_gaw_t_so_detail_header_merge_0_1.Lyr1_Intg_BDB_gaw_t_so_detail_header_merge - A <tHDFSConfiguration_1> configuration component has been found.
[INFO ]: dwh.lyr1_intg_bdb_gaw_t_so_detail_header_merge_0_1.Lyr1_Intg_BDB_gaw_t_so_detail_header_merge - A <tHiveConfiguration_1> configuration component has been found.
[INFO ]: dwh.lyr1_intg_bdb_gaw_t_so_detail_header_merge_0_1.Lyr1_Intg_BDB_gaw_t_so_detail_header_merge - A <tSparkConfiguration_1> configuration component has been found.
[INFO ]: org.apache.spark.sql.internal.SharedState - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/hadoop/Talend-JobServer-20181026_1147-V7.1.1/TalendJobServersFiles/repository/DWH_Lyr1_Intg_BDB_gaw_t_so_detail_header_merge_20190220_184110_UTrPh/Lyr1_Intg_BDB_gaw_t_so_detail_header_merge/spark-warehouse').
[INFO ]: org.apache.spark.sql.internal.SharedState - Warehouse path is 'file:/home/hadoop/Talend-JobServer-20181026_1147-V7.1.1/TalendJobServersFiles/repository/DWH_Lyr1_Intg_BDB_gaw_t_so_detail_header_merge_20190220_184110_UTrPh/Lyr1_Intg_BDB_gaw_t_so_detail_header_merge/spark-warehouse'.
[INFO ]: org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@9e02f84{/SQL,null,AVAILABLE,@Spark}
[INFO ]: org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1e6060f1{/SQL/json,null,AVAILABLE,@Spark}
[INFO ]: org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@39023dbf{/SQL/execution,null,AVAILABLE,@Spark}
[INFO ]: org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1a2ac487{/SQL/execution/json,null,AVAILABLE,@Spark}
[INFO ]: org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@563ccd31{/static/sql,null,AVAILABLE,@Spark}
Unable to connect to localhost on the port 3807 

 

 

I am not sure why logs are shown..

 

Employee

Re: A lot of Messages shown in the Big Data Platform studio

Hi,

 

    It says there is connection error in some component.

 

     Could you enable only specific components in the flow and check whether its working? Also did you verify whether the data is coming in output area correctly?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog