Can we Author SQL functions in Talend Data Studio instead in Source Database?

Anonymous — Sun, 24 Mar 2019 18:34:21 GMT

One of the question the Dev team raise - Analytics functions needs to be closer or should be part of ETL rather than in DB. Do you agree and if yes Can we create functions in talend?

Re: Can we Author SQL functions in Talend Data Studio instead in Source Database?

DataTeam1 — Sun, 24 Mar 2019 20:49:49 GMT

You can create your own java function in Talend (right click on routines and select Create routine)

Re: Can we Author SQL functions in Talend Data Studio instead in Source Database?

vapukov — Mon, 25 Mar 2019 04:36:43 GMT

@balasista wrote:

One of the question the Dev team raise - Analytics functions needs to be closer or should be part of ETL rather than in DB. Do you agree and if yes Can we create functions in talend?

Hi,

there no single answer, all depends from many factors

for example:

count average above new incremental data (or any other function) - good to be in ETL because calculated "on the fly"
re-calculate aggregation over past 3 months when you add daily incremental portion - good to be in the database, because 99% of data is already there
and some of the functions already in the database and could be only there - like triggers
lookup for big input data over small reference table - good to be in ETL, but lookup over huge table best todo inside a database (staging table + code) after import.

you could mix technologies depending on your needs and expected benefits - you could have routines in the database, but call some of them from Talend. You will have some logic hidden from the ETL layer, but you will see what are routines involved.

regards, Vlad

topic Can we Author SQL functions in Talend Data Studio instead in Source Database? in Talend Studio

Can we Author SQL functions in Talend Data Studio instead in Source Database?

Re: Can we Author SQL functions in Talend Data Studio instead in Source Database?

Re: Can we Author SQL functions in Talend Data Studio instead in Source Database?