One Star

How to load Data from Oracle DB Tables into AMazon S3 using Talend?

HI Experts,
Basically i am new to Talend ETL Tool.
I have a requirement that i need to load the data from Oracle DB Tables into Amazon S3 using Talend D.I Tool.
I have seen the things we can load the files into Amazon S3 but how about the data loading data from Oracle DB Tables into Amazon S3 in talend etl tool.
Can anyone suggest me for the above query.
Appreciate for your help!!!!

Thansk & Regards,
Sudheer Reddy
20 REPLIES
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
For your requirement, the workflow should be: tOracleInput-->tFileOutPutDelimited-->onComponentOk-->tS3Put.
Give a variable file path ((String)globalMap.get("tFileOutputDelimited_1_FILE_NAME")) in tS3Put which save oracle DB table data and will be uploaded to the S3 server.
See my screenshot
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
Thanks For the reply.
Can you please let me know, what is ((String)globalMap.get("tFileOutputDelimited_1_FILE_NAME")) this mean?
And using this in File name option instead of the full path of the filename?

Thanks & Regards,
Sudheer Reddy
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
((String)globalMap.get("tFileOutputDelimited_1_FILE_NAME")) is a existed global variable which retains filename/path from tFileOutputDelimited component.(In this case, it should be like D:/Talend-Studio-r104014-V5.3.1/workspace/out.csv).
Please see related document about TalendHelpCenter:How to centralize contexts and variables.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi xdshi,
I'm new on Talend and I would like to query a Big Oracle Table by using partition name with variable
How I can do that.
thanks in advance.
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi dabdougadry,
Please open a new forum for your topic.
It would be way more efficient if you were opening a new thread because people having the same issue could then find it more easily.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

hi,

I want to fetch data from mysql table and insert it into vertica database with following conditions
* the job should be scheduled at certain interval
* the duplication of data should not happen(loading same data twice)
* how to keep track of last loaded data
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
the job should be scheduled at certain interval

The scheduler tool is only available in Talend Enterprise subscription product. With Talend Open Studio for Data Integration, you can export the job script, and then schedule it with third scheduler tool, such as crontab command on Linux(window scheduler).
the duplication of data should not happen(loading same data twice)

Check the key box of the corresponding columns on the schema of tVerticaOutput, and select 'insert or update' option in the action on data list, the job will insert the new records if the key does not exist in the target table, otherwise, update the record.tVerticaOutput.
how to keep track of last loaded data

Please have a look at component TalendHelpCenter:tMysqlLastInsertId.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

hi
Regarding the above topic ,is db to db faster or db to file to db works good.
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
Regarding the above topic ,is db to db faster or db to file to db works good.

Usually, it depends on your situation.
The followings aspects could affect the job performance:
1. The volume of data, read a large of data set, the performance will degrade
2. The structure of data, if there are so many columns on t<DB>Input, it will consume many memory and much time for transferring the data during the job execution.
3. The database connection, the job always runs better if the database is installed on local, if the database is on another machine, even you are on VPN, you may have the congestion and latency issues.

Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

hi sabrina

But insert or update is too slow and we dont have tverticalastinsertid since i cant use tmysqllastinsertid
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

hi

How to make job read only new data for example
table1
id val
1 a
2 b
3 c
First time job reads this from mysql db and inserts into vertica db
Next time
table1
id val
1 a
2 b
3 c
4 d
5 e
The job should read oly last two values from mysql and put it in vertica how to achieve this ?
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

hi

How to use inner join or outer join so that i can get
4 d
5 e
values in the output.
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
But insert or update is too slow and we dont have tverticalastinsertid since i cant use tmysqllastinsertid

What's the row rate? Could you share your workflow screenshot with us?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
How to use inner join or outer join so that i can get
4 d
5 e
values in the output.

You can inner join source table and target table in tMap.
The workflow should be: t<DB>input(target)-->main-->tMap-->output
t<DB>input(source)-->lookup-->
Set "Catch lookup inner join reject" as true. The processed output data should be the updated data.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

hi
I have configured the job on same lines as said above .
But inner join reject is not working for me the output i'm getting is data common in both tables.

t<DB>input(target)-->main-->tMap-->output
t<DB>input(source)-->lookup-->
columns are id and value i'm putting source table columns to output
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
Could you please show us your tMap editor screenshot? Did you set inner join?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

in tmap i have set inner join.
Even then its not coming only the data which is same in both tables is coming as output.
Moderator

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
Could you please show us your tMap editor screenshot so that we can address your issue quickly.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

  Hi Sabrina,Is there a way to spool the file directly from an RDBMS table to S3 without a need to store the file in intermediate location (D:/Talend-Studio-r104014-V5.3.1/workspace/out.csv)) ? i.e., OracleInput --> S3 bucket?
One Star

Re: How to load Data from Oracle DB Tables into AMazon S3 using Talend?

Hi,
I build a job with MySql server in Talend Studio. Could you please tell that how can I export the same job and run the same with oracle database in another machine which doesnt have Talend Studio

Thanks