Get the total count of the output of tMap(expression filter) to use in the following tMap component

Highlighted
Seven Stars

Get the total count of the output of tMap(expression filter) to use in the following tMap component

Hi,

 

How to pass the row count of tMap output to another tMap for hashcode mapping?

 

row1: I am reading a file using tFileInputExcel_1.

rowY and rowN: Using tMap expression filter, I am splitting input into 2 datasets based on the input field value('Y' or 'N')

row2: I have connected a tFlowMeter to get the count of the rowY

row3: Is the output of tFlowMeterCatcher.

I have connected tFlowMeterCatcher to tJavaRow component where I have assigned row3.count to globalMap table as below:

 globalMap.put("cnt_rowY", row3.count);

 

Now from tFlowMeter I have connected a tMap component to get the value of globalMap.get("cnt_rowY")

But this always gives me null.

 

I wanted the mod of : ((key.hashCode() & 0x7fffffff) % cnt_rowY). 

How can I achieve this.

 

Thanks


Accepted Solutions
Community Manager

Re: Get the total count of the output of tMap(expression filter) to use in the following tMap component

The problem you are having is that the all rows are not guaranteed to complete between components. However, you can put a hack in to force this to work. The steps are below...

1) In your first tMap (where you want the output count), use a column to output a count. That column should use some code similar to this...

routines.Numeric.sequence("count", 1, 1) 

This will provide a row count.

2) Next, link this to a tSortRow and set this to be sorted by the count column. Set the order to descending. Therefore the first row out of this component will hold the max count number.

3) In your next tMap, use a tMap variable (the box in the middle) and set the expression to something like below...

Var.count==null ? row3.count : Var.count 

What this does is assign the max count number to the Var.count variable IF it is null (which it will be for the first row). Then for subsequent rows the value will remain the same. This will be your record count. You need to make sure the "nullable" box is ticked so that the variable can be null. After you have done this, every row will hold the max number of rows.

 

It's a bit counter-intuitive, but what you are doing here is forcing all rows to be processed by the tSortRow component before they are passed on. You are reordering them here, which means your max count will be sent first to the next component. Then you are taking the first value of count and forcing that to be assigned to every row. 


All Replies
Community Manager

Re: Get the total count of the output of tMap(expression filter) to use in the following tMap component

The problem you are having is that the all rows are not guaranteed to complete between components. However, you can put a hack in to force this to work. The steps are below...

1) In your first tMap (where you want the output count), use a column to output a count. That column should use some code similar to this...

routines.Numeric.sequence("count", 1, 1) 

This will provide a row count.

2) Next, link this to a tSortRow and set this to be sorted by the count column. Set the order to descending. Therefore the first row out of this component will hold the max count number.

3) In your next tMap, use a tMap variable (the box in the middle) and set the expression to something like below...

Var.count==null ? row3.count : Var.count 

What this does is assign the max count number to the Var.count variable IF it is null (which it will be for the first row). Then for subsequent rows the value will remain the same. This will be your record count. You need to make sure the "nullable" box is ticked so that the variable can be null. After you have done this, every row will hold the max number of rows.

 

It's a bit counter-intuitive, but what you are doing here is forcing all rows to be processed by the tSortRow component before they are passed on. You are reordering them here, which means your max count will be sent first to the next component. Then you are taking the first value of count and forcing that to be assigned to every row. 

Seven Stars

Re: Get the total count of the output of tMap(expression filter) to use in the following tMap component

Thanks.

This solution worked.

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download