Using Stats, Logs and Meters at run time

One Star

Using Stats, Logs and Meters at run time

Hi guys
I have built a simple little job to read in a spreadsheet and load to an Oracle table (using TOS 3.04). Job uses spreadsheet, table and database connection meta data.
This works fine.
I then try to enable the stats, log and meter logging functionality. I want it to be written to databse tables, so I specify my database connection, and some names for the various tables. I have 2 questions:
1. Do I need to create these tables upfront, or will TALEND create this for me. If so, where do I find an example of what this DDL must look like?
2. When I try to run as is, I get databse connection errors, as shown below:
DBI connect('host=cimart.mweb.co.za;ORACLE_SID=EDWDEV;port=1521','Jthomas',...) failed: Can't connect using this syntax without specifying a HOST and one of SID INSTANCE_NAME SERVER SERVICE_NAME at C:\TOS-Win32-r22547-V3.0.4\workspace\.Perl\JULIAN.job_LD_LKP_POSTAL_CODES_0.1.pl line 1492
DBI connect('host=cimart.mweb.co.za;ORACLE_SID=EDWDEV;port=1521','Jthomas',...) failed: Can't connect using this syntax without specifying a HOST and one of SID INSTANCE_NAME SERVER SERVICE_NAME at C:\TOS-Win32-r22547-V3.0.4\workspace\.Perl\JULIAN.job_LD_LKP_POSTAL_CODES_0.1.pl line 1046
can't connect to database at C:\TOS-Win32-r22547-V3.0.4\workspace\.Perl\JULIAN.job_LD_LKP_POSTAL_CODES_0.1.pl line 1046.
Tags (1)
Community Manager

Re: Using Stats, Logs and Meters at run time

Hello Julian
1. Do I need to create these tables upfront, or will TALEND create this for me. If so, where do I find an example of what this DDL must look like?

You must create a table in database first, the fileds should be the same as tLogCatcher'schema.
2. When I try to run as is, I get databse connection errors, as shown below:

You should set the connection parameter, contains host, database name, username etc..see my screenshot.
Best regards
shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Using Stats, Logs and Meters at run time

Hi there
Thanks for the response. Will give this a bash. I assume I must create one for each, i.e. logs, stats and volumetrics?
Wouldn't it be a good idea to provide the user with the ability to generate these tables automatically if the table does not exist? Or at least to allow the user to click a button on the tlogcatcher to create table?
Community Manager

Re: Using Stats, Logs and Meters at run time

Hello
Thanks for the response. Will give this a bash. I assume I must create one for each, i.e. logs, stats and volumetrics?

Yes, because all of them have different schema, table structure.
Wouldn't it be a good idea to provide the user with the ability to generate these tables automatically if the table does not exist? Or at least to allow the user to click a button on the tlogcatcher to create table?

tlogcatcher is not a db component, so we can't provide a function of creating a table on it. To create a table, using tCreateTable component. Just give you a tip to define a table schem quickly, you can open the schema of tLogCatcher and export the schema structure to a xml file, then import the xim file as the schema strcture of tCreateTable.
Best regards
shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Using Stats, Logs and Meters at run time

Hi there
Ok, I did as you said, but I still get a strange error:
Starting job LD_MAP_BLOCK_SUBURB at 13:56 02/06/2009.
connecting to socket on port 3432 ...
Can't call method "prepare" on an undefined value at C:\TIS\TOS-Win32-r24830-V3.1.1\lib\perl/tOracleOutput/Oracle.pm line 150.
connected
Job LD_MAP_BLOCK_SUBURB ended at 13:56 02/06/2009.

As far as I can determine, I have done everything exactly as you said. In my job component tab, I set the log stats option on, I have selected "On Databases" and have used a repository database connection. I have even used the repo connection to browse for the exact table that I created. Everything looks pretty much the same as what you have done?
I would insert an image, but at the risk of sounding stupid, haven't figured out yet how to embed images in the this message window!
One Star

Re: Using Stats, Logs and Meters at run time

Hey there
Thanks for the advice on the tCreateTable component - thats gonna save me loads of timeSmiley Happy
Employee

Re: Using Stats, Logs and Meters at run time

I would insert an image, but at the risk of sounding stupid, haven't figured out yet how to embed images in the this message window!

168
One Star

Re: Using Stats, Logs and Meters at run time

Ok, the reason I was getting the error on "STAT" was because my scd component had log stats set, but I had not create the stats table yet. I have switched that off, so currently, I only have logs set, and I still get the original error:
Starting job LD_MAP_BLOCK_SUBURB at 14:09 02/06/2009.
connecting to socket on port 4041 ...
connected
Can't call method "prepare" on an undefined value at C:\TIS\TOS-Win32-r24830-V3.1.1\lib\perl/tOracleOutput/Oracle.pm line 150.
Job LD_MAP_BLOCK_SUBURB ended at 14:09 02/06/2009.
Employee

Re: Using Stats, Logs and Meters at run time

This topic looks a lot like 6777 and you julian, you didn't answer to my question :-/
I strongly encourage you to become a member on the forum (so that you can get notification on replies in your topics)
One Star

Re: Using Stats, Logs and Meters at run time

My apologies guys
I had forgotten my password and was to lazy to sort this out! All sorted, now I will try to post some screen shots for u - sorry for the hassle!
One Star

Re: Using Stats, Logs and Meters at run time

Here are the images
One Star

Re: Using Stats, Logs and Meters at run time

I am starting to really question whether this functionality works at all!
I just changed the project settings to first write to console and that didn't work...no results. I then changed the project settings to write to files on my computer instead....that did not work either. No files are created. I have double checked that project settings have filtered down to job, see attachments.
So it does not seem as though any stats, logs or volumetrics are being generated at all.....does this functionality actually work in TOS 3.1.1?
Employee

Re: Using Stats, Logs and Meters at run time

When I ask statistics on files, it works perfectly as expected. but I reproduce a problem with statistics on database (I've tested with MySQL). I'm investigating.
Employee

Re: Using Stats, Logs and Meters at run time

After a deeper investigation, I confirm there was a bug introduced (certainly by 6524) to Perl + Stats&Logs + database. The idea was to have as few database connections as possible.
Did you know that "Stats&Logs" features is in fact hiding a tStatCatcher + tMysqlOutput? The problem comes from the way the hiden components are configured, but if you don't use "Stats&Logs" and have explicit tStatCatcher in your job, then the problem does not exist.
I'm working on a clean fix.