Five Stars

How to process files only for the past hour?

I have continuous sensor data coming in every 5 mins in form of files. I want to pick files only for the past hour and do the required processing. for e.g: talend job runs at 12:01pm , it should pick all the files from 11:00 am to 12:00 pm only.
Can anyone please suggest the approach I should take to make this happen within talend.

5 REPLIES
Moderator

Re: How to process files only for the past hour?

Hi,

Could you please elaborate your case with an example with input and expected output values?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Ten Stars

Re: How to process files only for the past hour?

if file name include date time pattern, just use

tFileList -> iterate

 

in tFileList, use pattern like:

"commonpart-*"+TalendDate.formatDate("yyyyMMddHH",TalendDate.addDate(TalendDate.getCurrentDate(),-1,"HH"))+"*.csv"

if file names not include time pattern - need work with tSystem and bash, look there

https://stackoverflow.com/questions/26968159/list-of-files-modified-1-hour-before

 

than parse result

-----------
Five Stars

Re: How to process files only for the past hour?

I had initiated a post on https://stackoverflow.com/ .

https://stackoverflow.com/questions/45550213/how-to-process-files-only-for-the-past-hour-using-talen...

 

I have reached to a point but facing issues(in comments)

Ten Stars

Re: How to process files only for the past hour?

no comments in thread (as I can see - You are move it to chat)

 

but in any case - suggested solution, no cover previous Hour,

If You check 1 Hr from current moment -  it could have or duplicates, or holes. You never can run Job exactly every Hr 

 

so You can mix recommendations and check not getCurrentDate() -1hr, but exactly low and high margins

 

more than 11:00:00 and less than 12:00:00

-----------
Five Stars

Re: How to process files only for the past hour?

Thanks Vapukov, currently i am going with getCurrentDate() -60mm.

 

Please suggest how can i specify 11:00:00 and less than 12:00:00. taking consideration that the job runs between the hourly intervals. 

 

e.g: if the job runs at 11.20 then it should pick the lower limit as 11 and upper limit as 12