One Star

Lack of records in Talend Log Table after die on error on component

Hello Talend Forge people,
I have a problem with generic Talend ODI mechanism.
Yesterday we had some technical problems and I wanted to diagnose it today.
However I spot that I don't have data in 'stats&logs' table or log file.
while I have records in stats table and in my additional error handler table there is no date in file nor database table created for talend logs:
select * from talend_logs where root_pid ='cJV8iN'
--no results
select * from talend_stats where root_pid ='cJV8iN'
moment              | pid    | father_pid | root_pid | system_pid | project      | job            | job_repository_id       | job_version | context | origin        | message_type | message | duration
--------------------+--------+------------+----------+------------+--------------+----------------+-------------------------+-------------+---------+---------------+--------------+---------+---------
2016-02-28 12:00:07 | cJV8iN | cJV8iN     | cJV8iN   |       1204 | | | _iUm4MM56EeWjBrdTJAQcTQ | 0.1         | Default |               | begin        |         |        
2016-02-28 12:36:33 | cJV8iN | cJV8iN     | cJV8iN   |       1204 | | | _iUm4MM56EeWjBrdTJAQcTQ | 0.1         | Default | tLogCatcher_1 | begin        |         |        
2016-02-28 12:36:36 | cJV8iN | cJV8iN     | cJV8iN   |       1204 | | | _iUm4MM56EeWjBrdTJAQcTQ | 0.1         | Default | tLogCatcher_1 | end          | success |     3052
2016-02-28 12:36:38 | cJV8iN | cJV8iN     | cJV8iN   |       1204 | | | _iUm4MM56EeWjBrdTJAQcTQ | 0.1         | Default |               | end          | failure |  2191137


select * from talend_errors where external_id_1 ='cJV8iN' 
id     | when_created        | external_id__1_type | external_id_1 | external_id__2_type | external_id_2  | external_id__3_type | external_id_3  | is_handled | source_job     | error_code         | error_message                                                                    | comments
-------+---------------------+---------------------+---------------+---------------------+----------------+---------------------+----------------+------------+----------------+--------------------+----------------------------------------------------------------------------------+---------
956621 | 2016-02-28 11:36:36 | pid                 | cJV8iN        | type                | Java Exception | origin              | tRedshiftRow_5 | false      | | unhalted_exception | org.postgresql.util.PSQLException:ERROR: Disk Full                               |        
                                                                                                                                                                                                   :   Detail:
                                                                                                                                                                                                   :   -----------------------------------------------
                                                                                                                                                                                                   :   error:  Disk Full
                                                                                                                                                                                                   :   code:      
                                                                                                                                                                                                   :   context:   node: 2
                                                                                                                                                                                                   :   query:    
                                                                                                                                                                                                   :   location:  fdisk_api.cpp:343
                                                                                                                                                                                                   :   process:   query2_74
                                                                                                                                                                                                   :   -----------------------------------------------

while for other jobs it work's without any problems: (because this is parent-child job I have to use pid instead of root_pid)
select * from talend_logs where pid ='BKgXiV';
moment              | pid    | root_pid | father_pid | project      | job                    | context | priority | type           | origin   | message                                                                          | code
--------------------+--------+----------+------------+--------------+------------------------+---------+----------+----------------+----------+----------------------------------------------------------------------------------+-----
2016-02-11 20:17:07 | BKgXiV | wtGYvY   | wtGYvY     | | jobnameexample| Default |        6 | Java Exception | tS3Put_1 | com.amazonaws.AmazonClientException:Unable to calculate MD5 hash: D:\xxxxx.csv (The system cannot find the file specified) |    1


select * from talend_stats where pid ='BKgXiV';
moment              | pid    | father_pid | root_pid | system_pid | project      | job                    | job_repository_id       | job_version | context | origin         | message_type | message | duration
--------------------+--------+------------+----------+------------+--------------+------------------------+-------------------------+-------------+---------+----------------+--------------+---------+---------
2016-02-11 20:16:53 | BKgXiV | wtGYvY     | wtGYvY   |       1016 | | | _GFsBENBLEeWOhMSUBpVjIQ | 0.1         | Default |                | begin        |         |        
2016-02-11 20:17:00 | BKgXiV | wtGYvY     | wtGYvY   |       1016 | | | _GFsBENBLEeWOhMSUBpVjIQ | 0.1         | Default | tRedshiftRow_3 | begin        |         |        
2016-02-11 20:17:00 | BKgXiV | wtGYvY     | wtGYvY   |       1016 | | | _GFsBENBLEeWOhMSUBpVjIQ | 0.1         | Default | tRedshiftRow_3 | end          | success |      515
2016-02-11 20:17:01 | BKgXiV | wtGYvY     | wtGYvY   |       1016 | | | _GFsBENBLEeWOhMSUBpVjIQ | 0.1         | Default | tS3Put_1       | begin        |         |        
2016-02-11 20:17:01 | BKgXiV | wtGYvY     | wtGYvY   |       1016 | | | _GFsBENBLEeWOhMSUBpVjIQ | 0.1         | Default | tS3Put_1       | end          | failure |      562
2016-02-11 20:17:02 | BKgXiV | wtGYvY     | wtGYvY   |       1016 | | | _GFsBENBLEeWOhMSUBpVjIQ | 0.1         | Default | tLogCatcher_1  | begin        |         |        
2016-02-11 20:17:02 | BKgXiV | wtGYvY     | wtGYvY   |       1016 | | | _GFsBENBLEeWOhMSUBpVjIQ | 0.1         | Default | tLogCatcher_1  | end          | success |      594
2016-02-11 20:17:10 | BKgXiV | wtGYvY     | wtGYvY   |       1016 | | | _GFsBENBLEeWOhMSUBpVjIQ | 0.1         | Default |                | end          | failure |    17219


select * from talend_errors where external_id_1 ='BKgXiV' ;
id     | when_created        | external_id__1_type | external_id_1 | external_id__2_type | external_id_2  | external_id__3_type | external_id_3 | is_handled | source_job             | error_code         | error_message                                                                    | comments
-------+---------------------+---------------------+---------------+---------------------+----------------+---------------------+---------------+------------+------------------------+--------------------+----------------------------------------------------------------------------------+---------
911290 | 2016-02-11 19:16:54 | pid                 | BKgXiV        | type                | Java Exception | origin              | tS3Put_1      | false      | | unhalted_exception | com.amazonaws.AmazonClientException:Unable to calculate MD5 hash: D:\xxxxx.csv (The system cannot find the file specified) |        


Do you have any idea what could went wrong and cause lack of data?
4 REPLIES
Moderator

Re: Lack of records in Talend Log Table after die on error on component

Hi,
What does your whole work flow look like? Did you use tStatCatcher and tLogCatcher components both in your parent and child job?
More information about your current job situation will be helpful for us to diagnose your issue.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Lack of records in Talend Log Table after die on error on component

Hello Sabrina, we have two cases
for NOT WORKING scenario:

We are using tLogCatcher component only to store additional information in our db structure.
In this case its stand alone job.

2) For working correctly scenario we are using tLogCatcher components in both child and parent jobs.
and logging mechanism defined in Stats&Logs tab
One Star

Re: Lack of records in Talend Log Table after die on error on component

Hello Sabrina,
I was told that we have some jobs with incomplete stats.
I have a strange example:
select * from talend_stats where root_pid ='swt1XT' order by moment;
moment	pid	father_pid	root_pid	system_pid	job_repository_id	job_version	origin	message_type	message	duration
2016-02-08 14:54:44 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 begin
2016-02-08 14:55:03 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tMysqlInput_2 begin
2016-02-08 14:55:52 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tMysqlInput_2 end success 48926
2016-02-08 14:55:53 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tMysqlInput_4 begin
2016-02-08 14:58:22 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tMysqlInput_4 end success 148419
2016-02-08 15:12:06 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tRedshiftRow_3 begin
2016-02-08 15:12:06 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tRedshiftRow_3 end success 499
2016-02-08 15:12:08 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tS3Put_1 begin
2016-02-08 15:12:59 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tS3Put_1 end success 50831
2016-02-08 15:12:59 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tRedshiftBulkExec_1 begin
2016-02-08 15:14:17 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tRedshiftBulkExec_1 end success 78194
2016-02-08 15:14:19 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tS3Put_2 begin
2016-02-08 15:14:19 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tS3Put_2 end failure 587
2016-02-08 15:14:20 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tLogCatcher_1 begin
2016-02-08 15:14:20 swt1XT swt1XT swt1XT 6316 _nsBOsIMAEeW9iONWbYfBqg 0.1 tLogCatcher_1 end success 551

As you can see we had an error in  tS3Put_2 and this error is visible in logs table.
After that error catcher component is triggered (we can see additional row in our own error table). Ends with success and.. nothing.
there is no row with swt1XT end status (and it should be 'failure').
Could you tell me if there is known scenario when we shouldn't expect end status info?
KR,
Michal
One Star

Re: Lack of records in Talend Log Table after die on error on component

Hello Sabrina, 
Can I count for any feedback?