Filename from global var

Highlighted
One Star

Filename from global var

Hi All.

 

I want to split a giant csv file into several smaller files according to the first three characters in the row. I have the following:

tFileInputFullRow --(row1)--> tJavaRow --(row2)-->tFileOutputRaw

     * rFileInputFullRow reads each line into a "line" column

     * tJavaRow reads:

       globalMap.put("rowType", row1.line.substring(1,4));
       output_row.line = input_row.line;

    * tFileOutputRaw has as filename "path/to"+((String)globalMap.get("rowType"))+".csv"

 

All I get as a result is a null.csv file staring back at me.

 

However, when I do:

tFileInputFullRow --(row1)--> tMap -(row2)-> tFlowToIterate --(iterate)-->tFixedFlowInput-->tFileOutputDelimited

with

* tMap adding a new column type to row2 which is defined just as in the tJavaRow above

* tFileOutputDelimited has the same name as tFileOutputRaw.

 

This time I do get the different files created!!

 

Why does this happen? I'm asking this because I'm seeing that the first solution goes much quicker than the latter (mainly because it doesnt have to iterate each of the 50 columns for each of the 600.000+ rows).

 

cheers

David

 

 

 

 


Accepted Solutions
Sixteen Stars TRF
Sixteen Stars

Re: Filename from global var

You need to break the process with tFlowToIterate else the output filename is computed only once when the subjob starts. Using tFlowToIterate you can change the filename on the flight but the process will be less fast.

TRF

View solution in original post


All Replies
Sixteen Stars TRF
Sixteen Stars

Re: Filename from global var

You need to break the process with tFlowToIterate else the output filename is computed only once when the subjob starts. Using tFlowToIterate you can change the filename on the flight but the process will be less fast.

TRF

View solution in original post

One Star

Re: Filename from global var

Even though the global variable is changed? (I have added another tJavaRow just to print the value of the global var and the value effectively changes each row...)

Sixteen Stars TRF
Sixteen Stars

Re: Filename from global var

Yes, tFlowToIterate set a new value for each global variable each time the input row changes.

TRF

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 3

Read about some useful Context Variable ideas

Blog