very slow job... what's the reason?

One Star

very slow job... what's the reason?

please see the screenshot, it is taking too much time to process just 1 million rows at the rate of 22 rows/second... what could be the reason?
tr_incomingtransaction = main input, three columns: ID, HostURL, RefererURL
tglobalvar = saving these three into global variables to use them later for splitting etc...
tExtractRegexFields = splits URL into domain name
more or less the same case with other components.
One Star

Re: very slow job... what's the reason?

Wild guess : do you use a cursor on your tPostgreSQLInput (a checkbox in advanced options) ?
On tOracle component this option drastically improved perf
One Star

Re: very slow job... what's the reason?

Try the below approach:
First you have to find out the component which is causing the issue. You can try this by deactivating the individual compoents and probably writing to tfileoutputdelimited.
Once you know the component which is causing the issue, we should work on it.
Seven Stars

Re: very slow job... what's the reason?

Possibly the problem is that each row5 is triggering the row8 lookup. I'd start by deactivating the second subjob.
Your whole job design looks unusual, like it could do with some significant optimisation. Some comments:
1. The data_warehouse connection component should be connected to tr_incoming_transaction not dim_searchengine.
2. tJava_2 should not have rows connected to it.
3. tSetGlobalVar_2 should be redundant as tFlowToIterate_1 populates global variables.
4. Similarly, what tSetGlobalVar_3 does could be done in tJavaRow_1.
One Star

Re: very slow job... what's the reason?

Thanks guys.. i think it's tFlowToIterate, which is taking more time and decreased the speed by 10s of times.
Any alternative solution to tFlowToIterate?
Seven Stars

Re: very slow job... what's the reason?

Depends what you're trying to achieve; you'll have to explain in more detail what each part of the job does.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Downloads and Trials

Test drive Talend's enterprise products.

Downloads