You need to join the current table row to the next row for each group key. Using a tMap component, you create a lookup table that shifts the main table one row at a time. When running the Job, an out of memory error occurs on the tMap component.
The lookup table size is too big. The idea to refer next row values is not appropriate.
Control break processing using a tMemorizeRows component that memorizes group key and columns on the previous row can solve this issue.
Increasing memory usage with JVM setting might solve this issue, but it is expensive in terms of RAM usage.
Using the Store on disk option in the tMap component might solve this issue. However, even if you use this option, it is inevitable that you’ll have to create a big lookup table, and that takes more time.
Implementing control break processing, by using temporary variables in the tMap component might solve this issue. However, it is difficult to read and understand. Moreover, it can memorize only one previous row while the tMemorizeRows component can memorize more previous rows.
The demonstration Job is available in the ControlBreakProcessing.zip file attached to this article.
In this example, the input data is grouped by group key and sorted in reverse order by sequence id.
The tMemorize component memorizes group key and columns on the previous row.
The tMap or tJavaRow components can join the current row to the previous row using a tMemorizeRows component. When the current group key differs from the previous group key, the row must be the first row for the group key, and it should be set to null because it has no previous row values.
The tFilterRow component filters the first row for each group key without previous row values.
The tSortRow component can set the order back to ascending.
... View more