One Star

[resolved] How to mark Files after processing

Hello,
In my job I have to process several csv files and I need a solution to keep track of which files in a folder have already been processed. Is there a way to sort of tag the files as processed or is it necessary to move/delete the files after processing?
In my scenario I have a folder for every Month of the year containing csv files with data of a week. I want a job that determines every file that have not been processed.

Thanks for your Help!
1 ACCEPTED SOLUTION

Accepted Solutions
Employee

Re: [resolved] How to mark Files after processing

Sadly I'm pretty busy today, but for the log file creation you would have something like that:

....> Last Component of the process ----> OnComponentOK ---> tFixedFlowInput (With the name of the current file in one column) --> tFileOutputDelimited (the log file)
so that would be :

tFileInputDelimited(theLogFile)
|
|
v
tFileList --> tFixedFlowInput (One column, the file name) -------> tMap ---(Get the Rejects of the inner join)
|
|> tFlowToIterate ---> tFileInputDelimited --->... Last Component of the process
|> OnComponentOK
|--->
tFixedFlowInput (With the name of the current file in one column)
|--->
tFileOutputDelimited (the log file)
For the filepaths, look at the variables created by the tFileList component (look at the list on the bottom left of the studio, and expand tFileList)
4 REPLIES
Employee

Re: [resolved] How to mark Files after processing

Hi,
I would just create a log file, and during the processing write the name of the processed csv files into that log file, so that in the next execution you just filter out the files already processed. Something like that:
tFileInputDelimited(theLogFile)
|
|
v
tFileList --> tFixedFlowInput (One column, the file name) -------> tMap ---(Get the Rejects of the inner join)---> tFlowToIterate ---> tFileInputDelimited --->...

There might be an easier way to do this, but these are my first thoughts.
One Star

Re: [resolved] How to mark Files after processing

Hi,
this seems like a good solution. I have realised your proposed job design, but I am struggling when it comes to creating and maintaining the log file.
Could you show me a sample job of how to create the log File and fill in the Filepaths?
Regards
Employee

Re: [resolved] How to mark Files after processing

Sadly I'm pretty busy today, but for the log file creation you would have something like that:

....> Last Component of the process ----> OnComponentOK ---> tFixedFlowInput (With the name of the current file in one column) --> tFileOutputDelimited (the log file)
so that would be :

tFileInputDelimited(theLogFile)
|
|
v
tFileList --> tFixedFlowInput (One column, the file name) -------> tMap ---(Get the Rejects of the inner join)
|
|> tFlowToIterate ---> tFileInputDelimited --->... Last Component of the process
|> OnComponentOK
|--->
tFixedFlowInput (With the name of the current file in one column)
|--->
tFileOutputDelimited (the log file)
For the filepaths, look at the variables created by the tFileList component (look at the list on the bottom left of the studio, and expand tFileList)
One Star

Re: [resolved] How to mark Files after processing

It works!
Thank you very much for your Help!