How to transform a job using many tMap to following subjobs?

One Star

How to transform a job using many tMap to following subjobs?

Hello,

I am using a job with a lot of following tMap (I have reduced the number to 3 tmap in the following picture 1).

My input file and output file are the same (identical path), that why I am using a temporary file and then I copy it OnSubJobOk.

I want separate each tMap and put it in a subJob (like in picture 3)
On each subJob I am using a temporary file and I copy it OnSubJobOk.

Then I call each subJob in my main job (see picture 2)

Could you tel me if it is the best way to do that? (using temporary csv files, following subJobs, ....)
One Star

Re: How to transform a job using many tMap to following subjobs?

it REALLY depends on what exactly you are doing and more important WHY .
Your job with 3 tMaps reads your source file only once. your new design would read it in each subjob, so that might be a disadvantage...depending on your precise problem.
Do you know the way data flows in talend?

Lordan
One Star

Re: How to transform a job using many tMap to following subjobs?

The reasons are:
1) I encountered the error problem saying "The code of method tFileInputDelimited_1Process() is exceeding the 65535 bytes limit" because in some case I have 10 tMap

2) Each subJob corresponds to an Attribute List Mapping (the inputFile "Author" have 2 columns SOURCE_VALUE and TARGET_VALUE) and the upperJob is an Object handling.
The advantage is that a subJob can be re-use for another object which is also using the same Attribute List.

Lordan wrote:
Your job with 3 tMaps reads your source file only once. your new design would read it in each subjob, so that might be a disadvantage...

It is the problem I saw with the "copy" problem, it is not really performant.

Lordan wrote:
Do you know the way data flows in talend?

What do you mean?
One Star

Re: How to transform a job using many tMap to following subjobs?

I have replaced the subJobs by Joblets (I am using TIS) its avoid the csv temporary file manipulation but I have always the error "The code of method tFileInputDelimited_1Process() is exceeding the 65535 bytes limit" ...
One Star

Re: How to transform a job using many tMap to following subjobs?

you have to split the main job to subjobs.
One Star

Re: How to transform a job using many tMap to following subjobs?

ashu wrote:
you have to split the main job to subjobs.

Yes, it is what I did (it is the title of my post ... ) but using temporary files is not really performant, and my question is how to use temporaries flows between subjobs ?
The usage of Joblets is a good way but the generated code is the same (same size) with or without joblet.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads