Migrate bigdata from source to target

Anonymous — Sat, 16 Nov 2024 09:33:00 GMT

hello

I have a project to migrate bigdata from a NoSQL source to a NoSQL target (both different servers) and i want to use talend.

does any one have a detailed document i can use

the user should be able to access the data on the target just like the source

this project is for archiving

thanks

Re: Migrate bigdata from source to target

Anonymous — Mon, 10 Jul 2017 03:35:36 GMT

Hi,

What's your source and target DB? Do you want to migrate data from hive to Hadoop? More information will be helpful for us to understand your current situation.

Best regards

Sabrina

Re: Migrate bigdata from source to target

Anonymous — Mon, 10 Jul 2017 07:16:16 GMT

Hi

My source is Oracle NoSQL and target is also OracleNoSQL

My plan is just to move 50% of the data from source to target so I can improve performance at source.

Target will also be a live database

Re: Migrate bigdata from source to target

Anonymous — Thu, 27 Jul 2017 16:11:44 GMT

Sorry my response was not very accurate and coming in later

source is hadoop and target is oracle database

is this possible and is thee a documentation

thanks

Re: Migrate bigdata from source to target

Anonymous — Fri, 28 Jul 2017 07:34:49 GMT

Hello,

Here is a component tSqoopExport which calls sqoop to transfer data from the Hadoop Distributed File System (HDFS) to a relational database management system (RDBMS).

Please take a look at this component reference:https://help.talend.com/reader/4p8tIcC_ZikiBxkRHfEfjQ/yfESEWoETjTda35UXJOCaw

Best regards

Sabrina

Re: Migrate bigdata from source to target

Anonymous — Sat, 29 Jul 2017 23:11:01 GMT

Thank you ,

I tried SqoopExport and got the following errors, where do I check the
logs to analyse and what could have caused this error

[statistics] connecting to socket on port 3966

[statistics] connected

[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable

[ERROR]: org.apache.hadoop.util.Shell - Failed to locate the winutils
binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in
the Hadoop binaries.

at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)

at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)

at org.apache.hadoop.util.Shell.(Shell.java:326)

at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)

at
org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)

at org.apache.hadoop.security.Groups.(Groups.java:77)

at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)

at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)

at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)

at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)

at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)

at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)

at
org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2554)

at
org.apache.hadoop.fs.FileSystem$Cache$Key.(FileSystem.java:2546)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2412)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(leapoc_sqoop.java:498)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(eapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(leapoc_sqoop.java:670)

[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in
the environment. Cannot check for additional configuration.

Note:
\tmp\sqoop-user\compile\1fcb574f77290c134ac7ed51fd28d25d\part10_sqoop.java
uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

[WARN ]: org.apache.sqoop.mapreduce.ExportJobBase - null FileStatus object
in isSequenceFiles(); assuming false.

[WARN ]: org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not
be able to find all job dependencies.

[WARN ]: org.apache.hadoop.mapreduce.JobSubmitter - Hadoop command-line
option parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.

[WARN ]: mapreduce.Counters - Group FileSystemCounters is deprecated. Use
org.apache.hadoop.mapreduce.FileSystemCounter instead

[WARN ]: mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter
is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead

[ERROR]: org.apache.sqoop.tool.ExportTool - Error during export: Export job
failed!

Exception in component tSqoopExport_1 (eapoc_sqoop)

java.lang.Exception: The Sqoop export job has failed. Please check the logs.

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(eapoc_sqoop.java:532)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(leapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(eapoc_sqoop.java:670)

[FATAL]: lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop - tSqoopExport_1 The Sqoop
export job has failed. Please check the logs.

java.lang.Exception: The Sqoop export job has failed. Please check the logs.

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.tSqoopExport_1Process(leapoc_sqoop.java:532)

at
lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.runJobInTOS(eapoc_sqoop.java:844)

at lo_eapoc.eapoc_sqoop_0_1.eapoc_sqoop.main(leapoc_sqoop.java:670)

[statistics] disconnected

Re: Migrate bigdata from source to target

Anonymous — Mon, 31 Jul 2017 07:40:54 GMT

Hello,

Did you configure the hadoop_home variable correctly?

Please take a look at this document about:TalendHelpCenter:The missing winutils.exe program in the Big Data Jobs

Let us know if it works.

Best regards

Sabrina

topic Re: Migrate bigdata from source to target in Talend Studio

Migrate bigdata from source to target

Re: Migrate bigdata from source to target

Re: Migrate bigdata from source to target

Re: Migrate bigdata from source to target

Re: Migrate bigdata from source to target

Re: Migrate bigdata from source to target

Re: Migrate bigdata from source to target