topic Talend Spark jobs using Dataframes in Talend Studio

Talend Spark jobs using Dataframes

nbang — Sat, 16 Nov 2024 07:40:21 GMT

Hi All,

I am new to Talend BigData. I am migrating all my DI jobs to Spark for faster execution.

I came across tSQLRow component which I read uses Spark SQL for execution. It was my observation that any operations like Join or aggregation worked faster using the tSQLRow against the components like tMap and tAggregateRow.

The only difference I could see was that Talend components work on RDDs where as tSQLRow works on Dataframes.

I was wondering if Talend components can also work on Dataframes instead of RDD.

Looking at current design I am almost moving every key based operation into tSQLRow. This is hampering the readability of my jobs.

Any comments regarding this would be appreciated.

Re: Talend Spark jobs using Dataframes

Jesperrekuh — Wed, 12 Sep 2018 14:52:57 GMT

Talend has generated java code by components, therefore somebody needs to implement the RDD or/and Dataframe functions, Both are totally different spark java api's ...
The answer is no... but yes if you / somebody is willing to modify the component and add a radiobutton * RDD * Dataframe.

Re: Talend Spark jobs using Dataframes

nbang — Fri, 14 Sep 2018 07:26:58 GMT

Do you mean that Talend is not handling RDDs even in Spark jobs ? I could see functions related to RDDs in the generated code. I could also see code related to Dataframes. However tMap deals with RDDs and tSQLRow deals with Dataframes.

Re: Talend Spark jobs using Dataframes

Jesperrekuh — Fri, 14 Sep 2018 11:33:22 GMT

No ... I mean there're some fundamental differences between them... and spark jobs will definitely handle them but it depends on how the components are constructed.

Different components : different strategies, the question is, which type (RDD, Dataframe, Dataset) is most appropriate to use : https://data-flair.training/blogs/apache-spark-rdd-vs-dataframe-vs-dataset/

Re: Talend Spark jobs using Dataframes

Viswa560 — Mon, 14 Sep 2020 11:19:59 GMT

Hi Team,

I am using Talend 7.3.1, Kindly let me know whether Talend using RDD's or Dataframes when I design a normal job with out Tsqlrow and by using Tmap, azure GEN2 , darabricks 5.5 LTS.

Thanks,

Viswa