I would like to know if there is a way to get file properties (like size, timestamp) of a file places in S3 bucket.
I have tried the following but unable to get file size and timestamps. Also does this differ if the file in S3 bucket is .gz file.
tS3FileList --iterate--> tFIleProperties --iterate--> tIterateToFlow --main--> tLogRow
So far, you have to retrieve your files from S3 and keep them on local firstly.
The work flow should be: tS3Connection-->tS3Get-->tFileProperties
As we have a client restriction not to download the file locally,canu please let us knw if there is a way other than downloading? or download/read file properties and delete in one component soo that we cant eve break this flow .ultimatley we shd nt have any control on seeing the file content.
I got the work around for this issue. It's using S3 Command line.
PFA job,component and console output images.
I tried to follow your approach, but I get an error...see the thread : https://community.talend.com/t5/Design-and-Development/unable-to-execute-command-using-tSystem-compo...
I have got same constraints. Cannot download the file locally.
We can use AWS CLI however then go off from using the components. As a suggestion can we have something in S3Get or a seperate component to read the file properties on S3?
Bumping this thread - has anything changed in the last 2 years? I can't see anything in the v7.0 docs.
I want to check to see if the files all have timestamps that are close together, i.e. there are no files left over from a previous day. There are hundreds of files and they are all many gigabytes so I can't download them!
Additionally, all of the suggestions of "download the file and check the timestamp" are no use since the tS3Get component gives the file a timestamp of the time the download was done, it does not replicate the S3 timestamp of the file onto the local file.
If your tS3List component is called, for example, tS3List_1, simply add the following entry to a Date column in the subsequent tIterateToFlow:
This is relying on undocumented internal variables. I have tested it in Talend v6.3 and 7.1. Another drawback is that if you copy and paste a pair of components, the globalMap.get("tS3List_1_CURRENT_KEY") call will be updated with the new component name, but the internal variable objectSummary_tS3List_1 will not be.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema