Five Stars

AWS s3 Filewatcher

Hi,

How to implement S3 filewatcher in talend..?

Thanks.

  • Data Integration
8 REPLIES
Seven Stars

Re: AWS s3 Filewatcher

do You want real watcher? ("real-time")

 

or just want from time to time check file exists or not?

-----------
Seven Stars

Re: AWS s3 Filewatcher

I suggest You check S3 notification system http://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html

 

I not test this way, but should be work - S3 -> SQS -> Talend 

-----------
Five Stars

Re: AWS s3 Filewatcher

just want from time to time check file exists or not?
Seven Stars

Re: AWS s3 Filewatcher

tS3BucketExists - if need check Bucket

tS3List - uncheck "List all object" checkbox and test patterns for objects

-----------
Five Stars

Re: AWS s3 Filewatcher

Hi  ,

I just want to check weather file is exist in s3 bukect or not around one hour means i thought to use the tsleep and tloop and iterate around one hour if file is not exist after one hour then we will populate the message file is not exist.. 

Seven Stars

Re: AWS s3 Filewatcher

You can use this components

 

Also - You can avoid tLoop and tSleep - just run You Job by scheduler. No reason to keep memory when it not necessary.

 

Other way (I use them some time) - install AWS cli on Talend Machine, and do this with command line interface with tSystem.

Example - I use this for fetch most resent backup from S3.

-----------
Five Stars

Re: AWS s3 Filewatcher

Other way (I use them some time) - install AWS cli on Talend Machine, and do this with command line interface with tSystem.

Example - I use this for fetch most resent backup from S3.

 

Hi ,

how to install  AWS cli on Talend Machine..

THanks.

Seven Stars

Re: AWS s3 Filewatcher

they have excellent docs on site, all what you need - keys which will need install when setup client
for this reason better to create separate for Talend

 

http://docs.aws.amazon.com/cli/latest/userguide/installing.html

 

http://docs.aws.amazon.com/cli/latest/userguide/installing.html

-----------