topic Re: How to replace two input database components by one? in Talend Studio

How to replace two input database components by one?

Anonymous — Sat, 16 Nov 2024 02:56:22 GMT

Hi there,

I'm trying to find a way to replace our two query input for one. At the moment we access the queries through tOracleInput and tHiveInput. Is there any component that can replace both?

At the moment I have only found tSqoopImportAllTables, but I have been unsuccessful so far...

Thank you in advance,

Tomas

Re: How to replace two input database components by one?

Anonymous — Thu, 19 Mar 2020 21:25:02 GMT

They are very different sources. Why would you want to use 1 component? Maybe I am misunderstanding your requirement?

Re: How to replace two input database components by one?

Anonymous — Fri, 20 Mar 2020 09:17:18 GMT

My goal is to, in someway, reduce the tables loading time. What happens at the moment is that, when i start running my job, it process one "Oracle input" first and after finishing goes to the next "Hive input", consuming time. I was thinking if instead of two "database input" components, I could use one component which could bring the two different databases, saving time.

Re: How to replace two input database components by one?

Anonymous — Fri, 20 Mar 2020 10:44:38 GMT

But you are using two data sources that work in different ways. If you were using two Oracle input components from the same database, it could be done by combining the queries and carrying out the join in the SQL. But since your sources are different, you cannot do this.

How much time is the data loading taking? Maybe there are other ways to improve performance. What is your job built to do?

Re: How to replace two input database components by one?

David_Beaty — Sat, 21 Mar 2020 15:22:34 GMT

Hi,

Depending on what version of Talend Studio (Open Source or Enterprise) you can parallelise the loads from Oracle and Hive to at least acquire the data at the same time.

Thanks