Can we Author SQL functions in Talend Data Studio instead in Source Database?

[Anonymous]

One of the question the Dev team raise - Analytics functions needs to be closer or should be part of ETL rather than in DB. Do you agree and if yes Can we create functions in talend?

[DataTeam1]

You can create your own java function in Talend (right click on routines and select Create routine)

[vapukov]

---

@balasista wrote:

One of the question the Dev team raise - Analytics functions needs to be closer or should be part of ETL rather than in DB. Do you agree and if yes Can we create functions in talend?

---

Hi,

there no single answer, all depends from many factors

for example:

• count average above new incremental data (or any other function) - good to be in ETL because calculated "on the fly" 
• re-calculate aggregation over past 3 months when you add daily incremental portion - good to be in the database, because 99% of data is already there
• and some of the functions already in the database and could be only there - like triggers 
• lookup for big input data over small reference table - good to be in ETL, but lookup over huge table best todo inside a database (staging table + code) after import.

you could mix technologies depending on your needs and expected benefits - you could have routines in the database, but call some of them from Talend. You will have some logic hidden from the ETL layer, but you will see what are routines involved. 

regards, Vlad