tS3Configuration in BigDataBatch Job is using s3a file system to enable inherit credentials from AWS role.
Is there any option to use S3N file system with inherit credentials option i.e; that is without providing access Key and secret Key ?
Problem is :
when we try to use tRedshiftConfiguration , it is required to provide tS3Configuration and when we select - use s3a file system, getting some access issues while running spark job in EMR using s3a file system option checked in talend s3configuration job.
java.nio.file.AccessDeniedException: s3a://<<Location>>/_temporary/0: innerMkdirs on s3a://<<location>>/_temporary/0: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;
Please reply - if anyone performed any redshift operations in spark job using s3a file system in s3configuration.
"Inherit credentials from AWS role" option is available in Big Data Spark Job (EMR) in 7.2.1, 7.1.2.
This means that it is not required to input access/secret key in tS3Configuration in Spark Batch and Streaming jobs.
Behind the scene the authentication used is the one of the EMR cluster / role so the job would have the same rights as the EMR cluster.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Read about OTTO's experiences with Big Data and Personalized Experiences
Pick up some tips and tricks with Context Variables
Take a look at the Talend Studio improvements for API Services