I have continuous sensor data coming in every 5 mins in form of files. I want to pick files only for the past hour and do the required processing. for e.g: talend job runs at 12:01pm , it should pick all the files from 11:00 am to 12:00 pm only.
Can anyone please suggest the approach I should take to make this happen within talend.
Could you please elaborate your case with an example with input and expected output values?
if file name include date time pattern, just use
tFileList -> iterate
in tFileList, use pattern like:
if file names not include time pattern - need work with tSystem and bash, look there
than parse result
I had initiated a post on https://stackoverflow.com/ .
I have reached to a point but facing issues(in comments)
no comments in thread (as I can see - You are move it to chat)
but in any case - suggested solution, no cover previous Hour,
If You check 1 Hr from current moment - it could have or duplicates, or holes. You never can run Job exactly every Hr
so You can mix recommendations and check not getCurrentDate() -1hr, but exactly low and high margins
more than 11:00:00 and less than 12:00:00
Thanks Vapukov, currently i am going with getCurrentDate() -60mm.
Please suggest how can i specify 11:00:00 and less than 12:00:00. taking consideration that the job runs between the hourly intervals.
e.g: if the job runs at 11.20 then it should pick the lower limit as 11 and upper limit as 12