spark configuration to use cluster created by tAmazonEMRManage

One Star

spark configuration to use cluster created by tAmazonEMRManage

Hi Experts, 
I am new to Talend and using Big data platform 6.1.1. I am able to create and launch a cluster using the TAmazonEMRManage however I wanted to use this as pre task to a big data batch job , using Amazon EMR Spark where I wanted to read from S3 and postgres (RDS) and write to S3 and postgres and I am facing following challenges
1> unable to pass the resource manager to the spark configuration of 2nd job dynamically.
2> unable to use 2 tS3Configuration components in same big data batch job, to read from multiple s3 buckets  
3> unable to find a postgres connector in big data batch job. 
Could you please advice . 
Thanks, 
ajmani 
Moderator

Re: spark configuration to use cluster created by tAmazonEMRManage

Hi,
Have you tried to use tS3XXX component in a standard job and call a spark job through subjob(tRunjob)?
For RDS, you can use spark component to achieve it
tMysql component for RDS(Aurora/Mysql), 
tOracle component for RDS(Oracle)
tJDBC component for RDS(MariaDB/PostgreSQL/SQLServer)
Let us know if it is Ok with you case.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Moderator

Re: spark configuration to use cluster created by tAmazonEMRManage

Hi ajmani ,
Is there any update for your issue?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: spark configuration to use cluster created by tAmazonEMRManage

Hi Sabrina,
Thanks for looking into this . You have suggested to use Tjdbc a spark component to connect to postgres RDS, but TJdbc components are not available in big data batch job,
Thanks, 
ajmani
Moderator

Re: spark configuration to use cluster created by tAmazonEMRManage

Hi,
Generic JDBC Component(tJDBC) in spark will be available in 6.2.
Here is the related jira issue:https://jira.talendforge.org/browse/PMBD-384
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now