topic Re: Data Integration Talend studio ElasticSearch ? in Talend Studio

Data Integration Talend studio ElasticSearch ?

Anonymous — Sat, 16 Nov 2024 09:30:32 GMT

Hello everyone

I am completely new to Talend, trying to learn the product, this is my first post here.

I have installed Data Integration Talend Open Studio version 6.4.1 on Windows 10 64-bit.

I need to get (i.e. read) data from ElasticSearch.

The Talend DI documentation PDF files which I downloaded do not mention ElasticSearch and there is no reference to tElasticSearch which I saw mentioned when I "googled" for a solution.

Can I use Talend DI (the "free version") to read/select/get/extract data from ElasticSearch ?

If so, could you give me some pointers to documentation or to other posts?

Thank you in advance

YuriB

Melbourne

Australia

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Wed, 19 Jul 2017 11:18:54 GMT

Hey YuriB, hope you're well.

I found some information on Google that this component is only available on Talend Open Studio for Big Data. Have you tried this one?

Thanks

Douglas

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Wed, 19 Jul 2017 14:35:17 GMT

Hi,

We have tElasticSearchInput and tElasticSearchOutput component in Big Data Spark Batch and Spark Streaming. You need to use a paid subscription Big Data or Realtime Big Data edition to get these components. We do not have ElasticSearch in traditional DI product.

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Thu, 20 Jul 2017 00:35:47 GMT

many thanks for your replies!

I will investigate Talend Big Data and the costs.

As an alternative would the following work, as I only have one single requirement to read data from ElasticSearch (buying Big Data may be too expensive for just one requirement):

Possible Solution using "free" Talend DI:

(1) write a set of Java methods/classes which will read data from ElasticSearch and expose that data to the caller (which may be any Java program).

(2) call those custom ElasticSearch Java functions/methods/classes from "free" Talend DI job which will pass the data read into the DI job for further processing

Please let me know if this is realistic solution?

If so, would this be complex/time consuming to write?

Would this perform well enough for high volume (5-10 million rows per hour)?

many thanks again!

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Thu, 20 Jul 2017 01:18:14 GMT

Hi,

Of course, you can potentially write this code and achieve connectivity with a Java API to ElasticSearch and Code Routines in Talend Studio. Your Code Routine will wrap the ElasticSearch API.

However, how complex will it be and how much time/effort it will take to write? I don't know. And remember about Avro Schemas etc that you may need to handle.

If you need to process 5 -10 million rows per hour, then your best approach is to do this in Spark Batch/Streaming. This way you can leverage the capability of a hadoop/spark cluster which can enable to meet this requirements.

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Thu, 20 Jul 2017 01:44:40 GMT

Hi,

It is available in paid version only. Not the opensource ones.

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Thu, 20 Jul 2017 01:57:08 GMT

many thanks!

are you aware of any blogs/videos/white-papers showing me how to write and integrate my own Java code into talend DI job?

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Thu, 20 Jul 2017 02:50:34 GMT

There are many examples. Look for tLibraryLoad, tJava, tJavaRow, tJavaFlex components example. You can also use code routine.

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Thu, 20 Jul 2017 02:52:07 GMT

thanks !!

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Thu, 20 Jul 2017 07:19:38 GMT

what about using the Talend ESB Open Studio which is also "free" (?????) and use cMessagingEndpoint which has elasticsearch available in it (Camel?) ??

Will this do the job and is this ESB "free" software ?

thanks

Re: Data Integration Talend studio ElasticSearch ?

Anonymous — Fri, 21 Jul 2017 08:39:11 GMT

in "free" Talend DI there is Palette: ESB, REST, tRESTClient which appears to work.

I dont know if this solution will perform but I am getting data returned from ElasticSearch via tRESTClient into Talend DI and sending it all to tLogRow, just as a test.