delimiter output file with increasing file name

Highlighted
Six Stars

delimiter output file with increasing file name

In this small example there is just a query which will export the values into a csv.

20-06-_2019_16-22-28.png

 

The output file is called "List0.csv".

How I can use a counter so that next time when the job will started again
the output file would look like this "List01.csv".

And next time again "List02.csv" etc.


I tried to use tJavaFlex and the following code

int count= Integer.valueOf(counter);
context.Counter=count+1;
System.out.println(count);


but it's not working.

Or can I set a global variable which will increase?

Thanks for some ideas/hints.


Accepted Solutions
Eight Stars

Re: delimiter output file with increasing file name

Hi @pawe84 

 

Whenever you start the job the global variables/context set earlier will be flushed.

 

Hence the only way is to store the job count in a file, and in the pre job read the file/Table, add one more to the job count and store to context/Global Variable and use the same context/global variable to name the file.

 

After that again update the file/table with new value of the counter.

 

Thanks and Regards,

Subhadip


All Replies
Eight Stars

Re: delimiter output file with increasing file name

Hi @pawe84 

 

Whenever you start the job the global variables/context set earlier will be flushed.

 

Hence the only way is to store the job count in a file, and in the pre job read the file/Table, add one more to the job count and store to context/Global Variable and use the same context/global variable to name the file.

 

After that again update the file/table with new value of the counter.

 

Thanks and Regards,

Subhadip

Six Stars

Re: delimiter output file with increasing file name

Now I tried to use tFileInputDelimited where I read a file with my star value.

Then the next component is tJavaRow with the following listing

 

context.count=input_row.seq;

context.count++;

 

But how I can get context.count now in order to save it tFileOutputDelimited? Which componend I have to use?

If I use tJavaRow again: output_row.seq=context.count it doesn't work I'll get an error.

 

Six Stars

Re: delimiter output file with increasing file name

Okay, now I got it.

I used tFixedFlowInput.

 

Is there a way tom reset the counter if a new day/ date is occuring?

E.g: Today the job was started five times --> stored value=5

 

When the job will be started tomorrow that there's a reset --> stored value=0 ? 

Employee

Re: delimiter output file with increasing file name

Hi,

 

   The best way is to store the date and running value to a database table (if not available, then use a control csv file).

 

    It should have two columns i.e., the date and current file sequence number. So at the beginning of your job, you will have to compare whether the date in the table(or file) is same as current date. If the condition is wrong, you can start the job with initial value i.e. 0. 

 

    If the condition for date is correct, you can assign the current value+1 to a context variable and use this value while creating the file name.

 

    Once the process is successfully completed, you need to insert both current date and current context value back to DB (or file) so that you can repeat the same process next time.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

    

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now