tS3Configuration in BigDataBatch Job is using s3a file system to enable inherit credentials from AWS role - is there any option to use S3N file system with inherit credentials option ?

Highlighted
Five Stars

tS3Configuration in BigDataBatch Job is using s3a file system to enable inherit credentials from AWS role - is there any option to use S3N file system with inherit credentials option ?

Hello:

tS3Configuration in BigDataBatch Job is using s3a file system to enable inherit credentials from AWS role.

Is there any option to use S3N file system with inherit credentials option i.e; that is without providing access Key and secret Key ?

 

Problem is :

when we try to use tRedshiftConfiguration , it is required to provide tS3Configuration and when we select - use s3a file system, getting some access issues while running spark job in EMR  using s3a file system option checked in talend s3configuration job.

 

Error:

java.nio.file.AccessDeniedException: s3a://<<Location>>/_temporary/0: innerMkdirs on s3a://<<location>>/_temporary/0: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

 

Please reply - if anyone performed any redshift operations in spark job using s3a file system in s3configuration.

 

Thanks

Vijay

Moderator

Re: tS3Configuration in BigDataBatch Job is using s3a file system to enable inherit credentials from AWS role - is there any option to use S3N file system with inherit credentials option ?

Hello,

"Inherit credentials from AWS role" option is available in Big Data Spark Job (EMR) in 7.2.1, 7.1.2.

This means that it is not required to input access/secret key in tS3Configuration in Spark Batch and Streaming jobs.

Behind the scene the authentication used is the one of the EMR cluster / role so the job would have the same rights as the EMR cluster.

Best regards

Sabrina

 

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend Studio Improvements for API Services

Take a look at the Talend Studio improvements for API Services

Watch Now