Long running job issue with Talend 6.2 enterprise edition/BigData

One Star

Long running job issue with Talend 6.2 enterprise edition/BigData

Hi, 
I am running a big data batch job(Map reduce) job with 58 million records, which is taking 5 hours to complete. There are few joins. The one thing I noticed that for every row, it is writing like below. So it is taking lot of time to complete.
Running job: job_1483713701028_0784
 map 0% reduce 0%
1|1|0.0|0.0                            <== This is getting printed in log, how to block this from printing??? 
1|1|0.0|0.0
1|1|0.0|0.0
1|1|0.0|0.0
1|1|0.0|0.0
1|1|0.0|0.0
Please note that I am reading a parquet file. I filtered the records and try to display only one record in the console, even then I am getting the above number of displays of "1|1|0.0|0.0"
I have increased Map and reduce memory to 5gb and 10GB. But not much improvement in reduction in the run time.
Question: 
1) Why is this getting printed? How to fix this? 
Appreciate your help!!!
One Star

Re: Long running job issue with Talend 6.2 enterprise edition/BigData

I did set log4jlevel to 'off' in the advance setting of the run tab. But still I see those lines displayed in the console. 
How did you solve these kind of issues? 
Please help me to solve this issue.
One Star

Re: Long running job issue with Talend 6.2 enterprise edition/BigData

The excess writing into log is fixed by introducing context variables. Earlier I had hard coded file paths.