delete files more than 45 days (example) old?

One Star

delete files more than 45 days (example) old?

So, I figure once I get a list of these files, I can just use tFileDelete to delete them. (I assume I would feed into tFileList and then make a connection to tFileDelete, like in the pic, but I'm not positive.)

What I don't know is how to read a directory and get a list of the (zip) files that are older that today minus x # of days.
Community Manager

Re: delete files more than 45 days (example) old?

Hello
The tFileProperties component(under File-->Management family) can fit your need. With tFileProperties component, you can get the create time of the file.
Best regards

shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: delete files more than 45 days (example) old?

what version is that in? Cause, its not showing up for me in TOS 2.3.3r13892

Ah, ok, I see it for the Perl version, but I'm in Java...
One Star

Re: delete files more than 45 days (example) old?

this bug topic (3545) seems to indicate that the java module is completed in 2.4.0RC1. are the fixes that are in 2.3.3 also in the latest 2.4.0 release? (RC2)

Guess I'm about to find out.
One Star

Re: delete files more than 45 days (example) old?

ok, so... is there somewhere I can find documentation on tFileProperties? cause its not in the most recent documentation available here http://www.talend.com/resources/documentation.php
and its not available in the help linked from TOS : http://127.0.0.1:50949/help/nav/2_2
or can someone tell me how to use this thing? please!
Community Manager

Re: delete files more than 45 days (example) old?

Hello
The tFileProperties component describes the properties of one file, include:
abs_path: the absolute path
dirname: the direcotry name
basename: the file name
mode_string: read or write
size: the size of file
mtime: the create time of file, return millisecond (long)
mtime_string: the create time of file, return a date (String)
Here is a simple scenario(see the screenshot)
Best regards
shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: delete files more than 45 days (example) old?

ok... now I can't figure out how I can filter on mtime_string and return either abs_path or basename to tFileList or tFileDelete
I mean, I think I can do some of the filtering in something like tMap, but I don't know if thats the right way to do it
Community Manager

Re: delete files more than 45 days (example) old?

Hello
ok... now I can't figure out how I can filter on mtime_string and return either abs_path or basename to tFileList or tFileDelete

Here is a Java example, Please see the screenshot.
 /**
* Returns the number of milliseconds since January 1, 1970, 00:00:00 GMT
* represented by this <tt>Date</tt> object.
*
* @return the number of milliseconds since January 1, 1970, 00:00:00 GMT
* represented by this date.
*/
public long getTime() {
return getTimeImpl();
}

Best regards

shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: delete files more than 45 days (example) old?

So I tried this process like you said, and it works perfectly... that is, until I try to change the # of days to anything greater than 24
I tried a bunch of different days, and all the ones <=24 worked, and all the ones >24 caused all the files to be deleted
Picture Information
1. contents of the folder before I run the clean up process
2. process with tFilterRow set to delete files created before 25 days
3. results of running the process defined in #2, where all files are deleted (whether they meet the filter criteria or not : before 5/6/08)
2. process with tFilterRow set to delete files created before 24 days
3. results of running the process defined in #4, where only the appropriate files are deleted (any files before 5/7/08)
Note:
tFileOutputDelimited_2 creates out_orig.txt (a list of all the files found in tFileList_3)
tFileOutputDelimited_1 creates out.txt (a list of all the files deleted by tFileDelete_1)
tFileOutputDelimited_3 creates out_notdeleted.txt (a list of all the files rejected by tFilterRow_1)

I find it kind of odd that the advanced filter in tFilterRow_1 works where ### (in the quoted formula below) is <=24, but not greater...
((TalendDate.getCurrentDate().getTime()-(table1.mtime+86400000*###))>0

One Star

Re: delete files more than 45 days (example) old?

Ok, I figured out how to get around the problem I posted above:
1. set ### (in the post above) to a context variable (ex: context.NumDays_Past)
2. set the Type of that context variable to Double
a. You could use Float or Long, in place of Double, here - they both worked for me
b. Don't use Integer or Short, they cause all the files in the folder to delete.
c. Don't use Big Integer, it causes an error (Exception in thread "main" java.lang.Error: Unresolved compilation problem: The operator * is undefined for the argument type(s) int, BigDecimal)

(TalendDate.getCurrentDate().getTime()-(86400000*context.NumDays_Past))>table1.mtime
One Star

Re: delete files more than 45 days (example) old?

Hi, I am having error to this context.NumDays_Past
nullpointerexception.
what values you put in context.NumDays_Past?
Regards.
One Star

Re: delete files more than 45 days (example) old?

Hi,I have reproduce the problem.
This is cause by 60*60*24*1000*25 out of scope of Integer type(result is: -2134967296)
So the result alway false
Solution:change 60*60*24*1000*25 ---to---> 60*60*24*1000*25L(result is: 2160000000)
Change the result is long type.
Four Stars RSH
Four Stars

Re: delete files more than 45 days (example) old?

Dear All,
I am trying to delete file which are 100 days old , I have used same approach which is mentioned in below email but I AM UNABLE TO UNDERSTAND FOLLOWING THINGS.
STEPS:
1) please check my log files which are in format of: (Name of the JOB_currentdate_time)
Example:==> Log_InsertDIMItems_2014-07-11-13-27-18
This file i have created on Friday 11th July
I want to delete files which are older than 100 days.
2) Screenshot of my job:
3) Code in tFilter Component: Advance Code:
(TalendDate.getCurrentDate().getTime()-(File_prop.mtime+86400000*24))>0
By using this query(24 days limit) then 0 files are deleted , but by using 25 all files are getting deleted which is confusing for me. Please advise where I am wrong ?
I am not getting how this is happening.
Can anyone please guide me on same.
My requirement is of deleting files which are 1oo days old .
Thanks and Regards
RSH
Four Stars

Re: delete files more than 45 days (example) old?

Hi RSH,
tFileList-->tFileProperties-->tJavaRow-->tFileDelete
Use this flow, assign file names to context variables in tJavaRow based on your date condition and use that context variable in tFileDelete component for file to delete..
try this
vaibhav
Four Stars RSH
Four Stars

Re: delete files more than 45 days (example) old?

Hi vaibhav,
Thank you for reply.
Its not possible to declare context for each file because we have near about 30 different names of files. Is there any possibility to manipulate on current snapshot which i provided?
I am not getting where I made mistake ?
Regards
RSH
Four Stars RSH
Four Stars

Re: delete files more than 45 days (example) old?

If I use :
(TalendDate.getCurrentDate().getTime()-(File_prop.mtime+86400000*25))>0
All files are getting DELETED (--NOT SURE HOW)
Please find below is an snapshot of result:
mtime mtime_string GetDate Calculated_Data_OF_Advance_Filter
1,40508E+12 Fri Jul 11 13:27:20 CEST 2014 1,40533E+12 2390162346
1,40508E+12 Fri Jul 11 13:27:20 CEST 2014 1,40533E+12 2390162867
1,40508E+12 Fri Jul 11 13:27:19 CEST 2014 1,40533E+12 2390163079
1,40508E+12 Fri Jul 11 13:27:20 CEST 2014 1,40533E+12 2390162220
1,40508E+12 Fri Jul 11 13:27:20 CEST 2014 1,40533E+12 2390162663

If i use
(TalendDate.getCurrentDate().getTime()-(File_prop.mtime+86400000*24))>0
NO file is deleted..
Four Stars

Re: delete files more than 45 days (example) old?

Hi,
mtime is unix timestamp in millisecond.. 86400000 represents one day
TalendDate.getCurrentDate().getTime()-86400000*2) - (input_row.mtime)>0
Above code deletes file which are older than two days.. you replace 2 by 100, then it should do the job... use wherever required in your design...
Try this.
vaibhav
Four Stars RSH
Four Stars

Re: delete files more than 45 days (example) old?

Hi Vaibhav,
I used the query which you suggested, but still strange result.
All files from server got deleted Smiley Sad
Not sure how this is working out:
Query:
(TalendDate.getCurrentDate().getTime()-86400000*100) - (File_prop.mtime)>0
Four Stars RSH
Four Stars

Re: delete files more than 45 days (example) old?

Hello ,
Can anyone please guide me on my issue please ?
Thanks in advance.
Regards
RSH
Five Stars

Re: delete files more than 45 days (example) old?

can you try below code. it will help you on this issue.
use this code in If condition.
new Date(File_prop.mtime).before(TalendDate.addDate(TalendDate.getCurrentDate(), -44, "dd"))==true

you can convert File_prop.mtime date into Date format using new Date(File_prop.mtime) constructor then you can add or minus days from date and compare with current date .

Re: delete files more than 45 days (example) old?

hi guys how are you.
i am Fabricio and i am from Brazil.
i am working in develop process for File delete Old.
figure the process, i am student of science of Computing.
In tMap:
Convert TimeStamp Unix for Date: new java.util.Date((long)(row1.mtime/1000*1000))

Five Stars

Re: delete files more than 45 days (example) old?

This is how I do it (I call it pruning).
I create some Variables in a tMap and then use Var.prune to determine if the file path should be emitted.
// Returns a long based on the oldest date you want to keep
((java.util.Date) globalMap.get("OldestDate")).getTime()
// Returns the number of milliseconds that the file (based on tFileProperties mtime) has passed it's pruning time.
Var.oldestMTime - archiveAllDirectories.mtime
// Returns true or false
// This value determines if the file path is output from a tMap.
Var.millisecondsPastPruning > 0 ? true : false 
This image shows a SubJob that collects the file names that need pruning. You can then iterate this to delete.