Not able to get all the files from S3 bucket. Any suggestions?

One Star

Not able to get all the files from S3 bucket. Any suggestions?

I'm trying to get all the files from the S3 bucket. I'm using tS3list and ts3get components. The tS3list component lists all the files in the bucket but the tS3get component is only getting the last file from the bucket that matches key prefix. I'm using TOS for Big Data Version: 6.0.0 version.
Thanks,
One Star

Re: Not able to get all the files from S3 bucket. Any suggestions?

Hi, try to set file path in ts3Get like this "file_path"+((String)globalMap.get("tS3List_1_CURRENT_KEY"))
Four Stars

Re: Not able to get all the files from S3 bucket. Any suggestions?

You are using tS3Get directly on tS3List onSubjobOk, which is giving the last object to tS3Get. You need to use "iterate" from tS3List to tS3Get to pass one by one object.

 

 

Four Stars

Re: Not able to get all the files from S3 bucket. Any suggestions?

Capture.PNG

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now