I would like to know if there is a way to get file properties (like size, timestamp) of a file places in S3 bucket.
I have tried the following but unable to get file size and timestamps. Also does this differ if the file in S3 bucket is .gz file.
tS3FileList --iterate--> tFIleProperties --iterate--> tIterateToFlow --main--> tLogRow
So far, you have to retrieve your files from S3 and keep them on local firstly.
The work flow should be: tS3Connection-->tS3Get-->tFileProperties
As we have a client restriction not to download the file locally,canu please let us knw if there is a way other than downloading? or download/read file properties and delete in one component soo that we cant eve break this flow .ultimatley we shd nt have any control on seeing the file content.
I got the work around for this issue. It's using S3 Command line.
PFA job,component and console output images.
I tried to follow your approach, but I get an error...see the thread : https://community.talend.com/t5/Design-and-Development/unable-to-execute-command-using-tSystem-compo...
I have got same constraints. Cannot download the file locally.
We can use AWS CLI however then go off from using the components. As a suggestion can we have something in S3Get or a seperate component to read the file properties on S3?
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema