tS3Configuration in BigDataBatch Job is using s3a file system to enable inherit credentials from AWS role.
Is there any option to use S3N file system with inherit credentials option i.e; that is without providing access Key and secret Key ?
Problem is :
when we try to use tRedshiftConfiguration , it is required to provide tS3Configuration and when we select - use s3a file system, getting some access issues while running spark job in EMR using s3a file system option checked in talend s3configuration job.
java.nio.file.AccessDeniedException: s3a://<<Location>>/_temporary/0: innerMkdirs on s3a://<<location>>/_temporary/0: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;
Please reply - if anyone performed any redshift operations in spark job using s3a file system in s3configuration.
"Inherit credentials from AWS role" option is available in Big Data Spark Job (EMR) in 7.2.1, 7.1.2.
This means that it is not required to input access/secret key in tS3Configuration in Spark Batch and Streaming jobs.
Behind the scene the authentication used is the one of the EMR cluster / role so the job would have the same rights as the EMR cluster.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema
Learn how to make your data more available, reduce costs and cut your build time