Six Stars

get total row transferred and processed by taland job?

Hi ,

 

  I am using sql server and oracle as a data source and want to dump data in cassendra. 

  I want to get and store the metadata of  the total rows transferred or inserted(similar to  the transferred row shown by talend while running the job) into the cassendra into another table,how can i do this ?? 

can it is possible by nb_line,tstatcatcher

1 ACCEPTED SOLUTION

Accepted Solutions
Twelve Stars

Re: get total row transferred and processed by taland job?

Ah sorry, I forgot to let you know something. You are kind of right in how you've used the code, but it has to be used after the component has finished. For example, you are trying to get the number of rows from your source DB component in a tMap connected to that component. That will not work because the source component won't have finished processing before the variable is used by the tMap. But, if you use that globalMap in a subjob that is run after the first subjob, you will find that the value is present. 

 

However, if you are wanting a row by row count, you can implement that pretty easily in a tMap using a Talend sequence. The following code will do it for you.....

routines.Numeric.sequence("mySeq", 1, 1)

 

 

Rilhia Solutions
4 REPLIES
Twelve Stars

Re: get total row transferred and processed by taland job?

You can do this using the built in AMC functionality. I talk about this here (https://www.rilhia.com/tutorials/talend-activity-monitoring-console-amc) and on linked tutorials from that tutorial. You will be interested in the FlowMeter stats. 

 

You can also write your own functionality and add those stats to your own tables. If you open a standard Talend job in the Studio, look in the bottom left corner of the application, you will see a section called "Outline". This shows a load of components you have used. If you expand each of those, you will see values which can be dragged and dropped into your job as Java code. They use the "globalMap" which holds a lot of useful values within Talend jobs. Look for values like "Number of line" and play with these within a tJava to see how they work.

Rilhia Solutions
Six Stars

Re: get total row transferred and processed by taland job?

Hi , Thanks for the reply, how can i use this outline in my talend jobs. I cannot drag them , even i tried this----

Twelve Stars

Re: get total row transferred and processed by taland job?

Ah sorry, I forgot to let you know something. You are kind of right in how you've used the code, but it has to be used after the component has finished. For example, you are trying to get the number of rows from your source DB component in a tMap connected to that component. That will not work because the source component won't have finished processing before the variable is used by the tMap. But, if you use that globalMap in a subjob that is run after the first subjob, you will find that the value is present. 

 

However, if you are wanting a row by row count, you can implement that pretty easily in a tMap using a Talend sequence. The following code will do it for you.....

routines.Numeric.sequence("mySeq", 1, 1)

 

 

Rilhia Solutions
Six Stars

Re: get total row transferred and processed by taland job?

Thanks, I implemented the first solution

 

Cap.PNG