HDFS2 external camel component

Anonymous — Sat, 16 Nov 2024 10:04:21 GMT

Hello,
I'm new in Talend ESB and i wonder how to load data into HDFS ?
My problem is that i get some information from web services and i need to load this data into hdfs, i found the camel component hdfs2 but i don't understand how to use it?
I have Talend DS 6.2.1 commercial distribution
Any help
Thanks

Re: HDFS2 external camel component

Anonymous — Thu, 23 Feb 2017 07:53:46 GMT

Hi,
Could you please try to create a new route like cMessagingEndpoint-->cLog
and configure the component cMessagingEndpoint like:
Basic setting: URI="hdfs2://192.168.xx.xxx:xxxx/user/automation/xxxx.csv"
Advanced setting: Dependencies= hdfs2 to see if this route works well?
Best regards
Sabrina

Re: HDFS2 external camel component

Anonymous — Thu, 23 Feb 2017 14:17:24 GMT

Hello Sabrina,
I have this error in my job:

No FileSystem for scheme: hdfs
java.io.IOException: No FileSystem for scheme: hdfs
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
	at org.apache.camel.component.hdfs2.HdfsInfo.<init>(HdfsInfo.java:36)
	at org.apache.camel.component.hdfs2.HdfsInfoFactory.newHdfsInfo(HdfsInfoFactory.java:31)
	at org.apache.camel.component.hdfs2.HdfsConsumer.setupHdfs(HdfsConsumer.java:81)
	at org.apache.camel.component.hdfs2.HdfsConsumer.doStart(HdfsConsumer.java:66)
	at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
	at org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:3234)
	at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:3528)
	at org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:3464)
	at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:3394)
	at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:3162)
	at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:3018)
	at org.apache.camel.impl.DefaultCamelContext.access$000(DefaultCamelContext.java:175)
	at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:2814)
	at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:2810)
	at org.apache.camel.impl.DefaultCamelContext.doWithDefinedClassLoader(DefaultCamelContext.java:2833)
	at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:2810)
	at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
	at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:2779)
	at org.apache.camel.main.Main.doStart(Main.java:126)
	at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
	at org.apache.camel.main.MainSupport.run(MainSupport.java:124)
	at local_project.test_hdfs_0_1.test_HDFS.run(test_HDFS.java:135)
	at local_project.test_hdfs_0_1.test_HDFS.runJobInTOS(test_HDFS.java:197)
	at local_project.test_hdfs_0_1.test_HDFS.main(test_HDFS.java:167)

I guess a need to add some librarys to my job but how can i configure an external camel component ?

Thanks for your help

topic HDFS2 external camel component in Talend Studio

HDFS2 external camel component

Re: HDFS2 external camel component

Re: HDFS2 external camel component