"GROUP BY" like in tMap ?

One Star

"GROUP BY" like in tMap ?

Is it possible to do following query feature with tMap component : GROUP BY with some calculation related on this grouping ?
One Star

Re: "GROUP BY" like in tMap ?

Stupid question or ... ?
Employee

Re: "GROUP BY" like in tMap ?

Your question is not stupid at all, it's just the opposite. The answer is "no". And some of us (me included) have been working on it this morning.

The processing you need will be available in a "tAggregate" component. This kind of component is special regarded to other existing components and not handled in the current "Perl code generation model" This kind of component is special because the first output row must wait for the last input row. Our discussions have lead us to the conclusion that we needed some improvement in the code generator engine. This requires more than a couple of days to be stable and releasable.

It's possible to simulate a tAggregate with the existing components (with a tHash and a tPerlFlex) but the solution is not very "visually understandable". Would you like me to show this solution or would you prefer to wait for the availability of the tAggregate?
One Star

Re: "GROUP BY" like in tMap ?

Not stupid at all!
What you describe is what we call tAggregate.

In fact you cannot simulate the tAggregate with a tMap.
tMap component work on a row by row basis. To perform calculation (for a tAggregate or even for a tSort), you need a component that will work on a complete data set (eg. you have to bufferize complete input before you can output any value) (I make it simple a little bit...).
Thus a tMap cannot siumlate this buffering behaviour.

We had a big discussion yesterday and this morning about tAggregate (and tSort) implementation with the R&D team and established a roadmap : 2 weeks seems to be enough to get those component working.

Regards,
Fabrice

PS : our tAggregate will support common calculations like : Sum, Max, Min, Avg, First, Last...and custom op and will support some optimization (memory, disk, group, sort...)
One Star

Re: "GROUP BY" like in tMap ?

Thanks for the answer (and the product).

I will wait for the stable version.
One Star

Re: "GROUP BY" like in tMap ?

I take it this component will essentially be a perl associative array (hash)?
Will you be able to configure setting on the component so that it can run in memory (for performance and smaller data loads) or be tied to a file in case of large data volumes?
One Star

Re: "GROUP BY" like in tMap ?

a tAggregate and a tSort, It's a good news !!

One request : a tFilter Component
Employee

Re: "GROUP BY" like in tMap ?

Hugo wrote:
I take it this component will essentially be a perl associative array (hash)?
Will you be able to configure setting on the component so that it can run in memory (for performance and smaller data loads) or be tied to a file in case of large data volumes?

You're perfectly right, tAggregate will first fill a hash and once no input left it will output its content.

The first release of the component will be "memory" only, but in a near future we'll have to provide an option to store hashes on disk. I've already tried DBM:Smiley Very Happyeep some months ago and was quite disappointed by low performances.
Employee

Re: "GROUP BY" like in tMap ?

favre wrote:
One request : a tFilter Component

Can you please create a new topic about "filtering"? Indeed, at Talend, we've started a discuss about a component dedicated to filtering and I would find useful to make this discussion public with users.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

APIs for Dummies

View this on-demand webinar about APIs....

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now