[resolved] Can we process Files From AWS S3 bucket without storing locally?

Highlighted
One Star

[resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi All,
Please could the experts let me know if when we connect to the AWS S3 bucket to process the files from it are they stored somewhere locally on the disk on which the job is run or is there any way to process the file without storing locally.
I am using the below components in the Job and the tS3Get component has the File option where we have to give the local path and the tFileInputDelimited to pick up from that location which we have given in the tS3Get component.
tS3Connection-->tS3List-->tS3Get-->tFileInputDelimited-->tMap-->tOracleOutput 
Thanks
Highlighted
Moderator

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi,
So far, there is no support for handling data by air. You have to get your files stored in S3 into local disk first then reading them.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
One Star

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Thanks Sabrina. Appreciate you response. Please could you also let me know if there are any components like tS3Input or tS3Output which can be used for this purpose of processing the files directly from the AWS S3 bucket without storing them to a local directory.
Thanks
Highlighted
Moderator

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi,
Could you please take a look at a jira issue:https://jira.talendforge.org/browse/TBD-929 to see is it what are you looking for?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
One Star

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi,
Yes , The component which I am looking for is probably the tS3Input which I am guessing can be used instead of the tFileInputDelimited when reading the files from AWS S3. Does it have the same functionalities like the tFileInputDelimited component?
Thanks
Highlighted
Moderator

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi,
So far, this feature jira issue is in Progress. There is no component reference about this component. We will keep you posted as long as the new component is available in Talend studio.

Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
One Star

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Thanks Sabrina. Appreciate the quick response.
Thanks
Highlighted
Five Stars

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hi Sabrina,

 

In the link you had shared earlier, I can see that this issue is closed. So, now is there ts3input component in TOS to get files directly from s3?

 

Thanks,

Alekhya.

Highlighted
Moderator

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

Hello @alekhyak 

We have supported for Talend MapReduce with S3 on Amazon EMR.

For more information, please have a look at this tS3Input component reference:https://help.talend.com/reader/Tr~X7jq25879O8pZsAQ1mg/Fc8uZPrpxI8fSCmmnK5qSA

Note: it is only for users who have subscribed to Talend Data Fabric or to any Talend product with Big Data but it is not applicable to Talend Open Studio for Big Data users.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
One Star

Re: [resolved] Can we process Files From AWS S3 bucket without storing locally?

How to extract data from csv file using ts3input or do we have any other way to read the data from csv file available in s3 bucket ?

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now