One Star

[resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi All,
Please could the experts let me know if when we connect to the AWS S3 bucket to process the files from it are they stored somewhere locally on the disk on which the job is run or is there any way to process the file without storing locally.
I am using the below components in the Job and the tS3Get component has the File option where we have to give the local path and the tFileInputDelimited to pick up from that location which we have given in the tS3Get component.
tS3Connection-->tS3List-->tS3Get-->tFileInputDelimited-->tMap-->tOracleOutput 
Thanks
6 REPLIES
Moderator

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi,
So far, there is no support for handling data by air. You have to get your files stored in S3 into local disk first then reading them.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Thanks Sabrina. Appreciate you response. Please could you also let me know if there are any components like tS3Input or tS3Output which can be used for this purpose of processing the files directly from the AWS S3 bucket without storing them to a local directory.
Thanks
Moderator

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi,
Could you please take a look at a jira issue:https://jira.talendforge.org/browse/TBD-929 to see is it what are you looking for?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi,
Yes , The component which I am looking for is probably the tS3Input which I am guessing can be used instead of the tFileInputDelimited when reading the files from AWS S3. Does it have the same functionalities like the tFileInputDelimited component?
Thanks
Moderator

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi,
So far, this feature jira issue is in Progress. There is no component reference about this component. We will keep you posted as long as the new component is available in Talend studio.

Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Thanks Sabrina. Appreciate the quick response.
Thanks