Six Stars

Migrate bigdata from source to target

hello

I have a project to migrate bigdata from a NoSQL source to a NoSQL target (both different servers) and i want to use talend.

does any one have a detailed document i can use

the user should be able to access the data on the target just like the source

this project  is for archiving

thanks

  • Big Data
6 REPLIES
Moderator

Re: Migrate bigdata from source to target

Hi,

What's your source and target DB? Do you want to migrate data from hive to Hadoop? More information will be helpful for us to understand your current situation.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Migrate bigdata from source to target

Hi


My source is Oracle NoSQL and  target is also OracleNoSQL  


My plan is just to move 50% of the data from source to target so I can improve performance at source.


Target will also be a live database



Six Stars

Re: Migrate bigdata from source to target

Hi

Sorry my response was not very accurate and coming in later

 

source is hadoop and target is oracle database

is this possible and is thee a documentation

 

thanks

Moderator

Re: Migrate bigdata from source to target

Hello,

Here is a component tSqoopExport which calls sqoop to transfer data from the Hadoop Distributed File System (HDFS) to a relational database management system (RDBMS).

Please take a look at this component reference:https://help.talend.com/reader/4p8tIcC_ZikiBxkRHfEfjQ/yfESEWoETjTda35UXJOCaw

Best regards

Sabrina

 
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Six Stars

Re: Migrate bigdata from source to target

Thank you ,

I tried SqoopExport and got the following errors, where do I check the
logs to analyse and what could have caused this error





[statistics] connecting to socket on port 3966

[statistics] connected

[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable

[ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils
binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in
the Hadoop binaries.

at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)

at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)

at org.apache.hadoop.util.Shell.(Shell.java:326)

at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)

at
org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)

at org.apache.hadoop.security.Groups.(Groups.java:77)

at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)

at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)

at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)

at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)

at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)

at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)

at
org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2554)

at
org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2546)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2412)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(leapoc_sqoop.java:498)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(eapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(leapoc_sqoop.java:670)

[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in
the environment. Cannot check for additional configuration.

Note:
\tmp\sqoop-user\compile\1fcb574f77290c134ac7ed51fd28d25d\part10_sqoop.java
uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

[WARN ]: org.apache.sqoop.mapreduce.ExportJobBase - null FileStatus object
in isSequenceFiles(); assuming false.

[WARN ]: org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not
be able to find all job dependencies.

[WARN ]: org.apache.hadoop.mapreduce.JobSubmitter - Hadoop command-line
option parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.

[WARN ]: mapreduce.Counters - Group FileSystemCounters is deprecated. Use
org.apache.hadoop.mapreduce.FileSystemCounter instead

[WARN ]: mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter
is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead

[ERROR]: org.apache.sqoop.tool.ExportTool - Error during export: Export job
failed!

Exception in component tSqoopExport_1 (eapoc_sqoop)

java.lang.Exception: The Sqoop export job has failed. Please check the logs.

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(eapoc_sqoop.java:532)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(leapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(eapoc_sqoop.java:670)

[FATAL]: lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop - tSqoopExport_1 The Sqoop
export job has failed. Please check the logs.

java.lang.Exception: The Sqoop export job has failed. Please check the logs.

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(leapoc_sqoop.java:532)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(eapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(leapoc_sqoop.java:670)

[statistics] disconnected

Moderator

Re: Migrate bigdata from source to target

Hello,

Did you configure the hadoop_home variable correctly?

Please take a look at this document about:TalendHelpCenter:The missing winutils.exe program in the Big Data Jobs

Let us know if it works.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.