is it possible to get outfiles with an incremental number?
Let's say we have a job and this job will create the file output_001.csv
If we start the job again then we'll get output_002.csv and if we start the job 3rd time then output_003.csv etc.
Thanks for any advice.
If you are storing the last file number in a location (it can be file, DB etc), you can read it from there to a context variable (say running_number).
While writing the file, you can add the file name in the output as "output_"+context.running_number+".csv"
Once the data output to file is complete, you can write the new output back to the area where you are storing last output file number. In this way, it will work fine. I would suggest to use a table in DB to store this control information.
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)
Below Approach would help
1) keep a File having last run sequence in your local disk . Read this file at the start of job and use it to create your next run file. Update the same file for next run sequence.
(Note : If you are using Context file , you can simply update new sequence in that file)
2) You could use system Variable instead of File . Get/Set using value tSystem and tSetenv
3) using tFileFetch , get the latest created file name in the output folder .
Get the last part from string so for output_003.csv it would be 003
Add 1 to it , use new value to create Output.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Take a look at this video about Talend Integration with Databricks
Learn how<SPAN>to modernize your Cloud Platform for Big Data Analytics with Talend and Microsoft Azure</SPAN>