Migrate bigdata from source to target

[Anonymous]

hello

I have a project to migrate bigdata from a NoSQL source to a NoSQL target (both different servers) and i want to use talend.

does any one have a detailed document i can use

the user should be able to access the data on the target just like the source

this project  is for archiving

thanks

[Anonymous]

Hi,

What's your source and target DB? Do you want to migrate data from hive to Hadoop? More information will be helpful for us to understand your current situation.

Best regards

Sabrina

[Anonymous]

Hi


My source is Oracle NoSQL and  target is also OracleNoSQL  


My plan is just to move 50% of the data from source to target so I can improve performance at source.


Target will also be a live database

[Anonymous]

Hi

Sorry my response was not very accurate and coming in later

source is hadoop and target is oracle database

is this possible and is thee a documentation

thanks

[Anonymous]

Hello,

Here is a component tSqoopExport which calls sqoop to transfer data from the Hadoop Distributed File System (HDFS) to a relational database management system (RDBMS).

Please take a look at this component reference:https://help.talend.com/reader/4p8tIcC_ZikiBxkRHfEfjQ/yfESEWoETjTda35UXJOCaw

Best regards

Sabrina

[Anonymous]

Thank you ,

I tried SqoopExport and got the following errors, where do I check the
logs to analyse and what could have caused this error





[statistics] connecting to socket on port 3966

[statistics] connected

[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable

[ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils
binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in
the Hadoop binaries.

at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)

at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)

at org.apache.hadoop.util.Shell.(Shell.java:326)

at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)

at
org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)

at org.apache.hadoop.security.Groups.(Groups.java:77)

at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)

at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)

at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)

at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)

at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)

at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)

at
org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2554)

at
org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2546)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2412)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(leapoc_sqoop.java:498)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(eapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(leapoc_sqoop.java:670)

[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in
the environment. Cannot check for additional configuration.

Note:
\tmp\sqoop-user\compile\1fcb574f77290c134ac7ed51fd28d25d\part10_sqoop.java
uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

[WARN ]: org.apache.sqoop.mapreduce.ExportJobBase - null FileStatus object
in isSequenceFiles(); assuming false.

[WARN ]: org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not
be able to find all job dependencies.

[WARN ]: org.apache.hadoop.mapreduce.JobSubmitter - Hadoop command-line
option parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.

[WARN ]: mapreduce.Counters - Group FileSystemCounters is deprecated. Use
org.apache.hadoop.mapreduce.FileSystemCounter instead

[WARN ]: mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter
is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead

[ERROR]: org.apache.sqoop.tool.ExportTool - Error during export: Export job
failed!

Exception in component tSqoopExport_1 (eapoc_sqoop)

java.lang.Exception: The Sqoop export job has failed. Please check the logs.

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(eapoc_sqoop.java:532)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(leapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(eapoc_sqoop.java:670)

[FATAL]: lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop - tSqoopExport_1 The Sqoop
export job has failed. Please check the logs.

java.lang.Exception: The Sqoop export job has failed. Please check the logs.

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(leapoc_sqoop.java:532)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(eapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(leapoc_sqoop.java:670)

[statistics] disconnected

[Anonymous]

Hello,

Did you configure the hadoop_home variable correctly?

Please take a look at this document about:TalendHelpCenter:The missing winutils.exe program in the Big Data Jobs

Let us know if it works.

Best regards

Sabrina