Seven Stars

Catch processed row's count

Hi there,

 

I'm using TDI open studio. I have designed and run the job, whose screenshot is :

 

InventoryWest.PNG

 

As you can see, there we are getting count of processed rows in green colour. I want to catch these count in separate file (prefer excel file).

 

So for above job, I expect below output :

op.PNG

 

 

After researching for a while, I come to know that there are 3 components that I feel one of these 3 can fulfill the requirement.

tStatCatcher

tLogCatcher

tFlowMeterCatcher

 

But I don't know which one to use and how to use it for my requirement. Please, help me out guys. 

 

Thank you Smiley Happy

 

  • Data Integration
Tags (4)
1 ACCEPTED SOLUTION

Accepted Solutions
Moderator

Re: Catch processed row's count

Hello,

The tFlowMeterCatcher component catches the processing volumetric from the tFlowMeter component and passes them onto the output component.

For example:
tFileInput--->tFlowMeter--->tMysqlOutput_1
tFlowMeterCatcher---tLogRow
There is one column called "count" in the schema of tFlowMeterCatcher which counts the no. of records pass by the specify flow.
Let me know if it is OK with you.

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
11 REPLIES
Moderator

Re: Catch processed row's count

Hello,

The tFlowMeterCatcher component catches the processing volumetric from the tFlowMeter component and passes them onto the output component.

For example:
tFileInput--->tFlowMeter--->tMysqlOutput_1
tFlowMeterCatcher---tLogRow
There is one column called "count" in the schema of tFlowMeterCatcher which counts the no. of records pass by the specify flow.
Let me know if it is OK with you.

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: Catch processed row's count

Hi @xdshi

 

Thank you so much. Your solution worked !

 

I tried 2 options :

 

1. Suggested by you :  using  tFlowMeterCatcher.- easy to record statistics in csv or excel file as it has output row.

 

2. Without using tFlowMeterCatcher - tried stats&log under job setting tab, where we check the box in front of 'Use volumetrics' and specify the file path and name. When we use this, we don't need to use tFlowMeterCatcher component. But its giving output file with values separated by ' ; ' (semicolon).

 

How can we get the the output that can be view properly as csv or excel using 2nd option ?

(might be comma separated, or using encoding, etc.)

 

volumetrics.PNG

 

Thanks Smiley Happy

 

Moderator

Re: Catch processed row's count

Hello,

We see that you have saved your 'Meter file name' as meter_file.csv

Could you please show us your actual output result with values separated by ' ; ' (semicolon)?

Please mask your sensitive data.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: Catch processed row's count

Please find below the screenshots of the output result :

 

opened in ms excel :

volumetric_output.PNG

 

opened in notepad :

valumetric_output_notepad.PNG

Moderator

Re: Catch processed row's count

Hello,

It should be field separator to separate fields for the transferred data when set the flow meter log to output the count record into a file.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: Catch processed row's count

Where is the option to set field separator ?
I can see only encoding option there in job setting-> stats&log -> use volumetrics -> on files.
is there anything by which we can get comma instead of semicolon ?
Moderator

Re: Catch processed row's count

Hello,

We will make an investigation on it and then come back to you as soon as we can.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: Catch processed row's count

Hi,

In tFlowMeterCatcher, we are not getting the duration of processing.
I come to know that there is component called tChronometerStop to get a duration. But its logging the time on console. How can I get this time in file ? Or is there workaround for the same ?

Moderator

Re: Catch processed row's count


AkshayNG wrote:
Hi,

In tFlowMeterCatcher, we are not getting the duration of processing.
I come to know that there is component called tFlowMeterCatcher to get a duration. But its logging the time on console. How can I get this time in file ? Or is there workaround for the same ?

If I understand your issue clearly, do you want to use component tStatCatcher in your work flow and log the output into file? If so, you can design your work flow like: tStatCatcher-->tFileOutPutxxx(your target output file).

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Seven Stars

Re: Catch processed row's count

Thanks @xdshi

 

I've tried tStatCatcher, and got the duration as well. But then I've to apply some lookup logic using tMap to get result of both in one output table containg row count (from tFlowMeterCatcher) and duration(using tStatCatcher).

 

I was basically looking for a way to add one more column for TIME  in result of tFlowMeterCatcher. This column must contain values as duration of the specified component.

Moderator

Re: Catch processed row's count

Hello,

So far, you cannot edit the schema setting of tFlowMeterCatcher component.

 

1.png

Best regards

Sabrina

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.