How to replace two input database components by one?
Hi there,
I'm trying to find a way to replace our two query input for one. At the moment we access the queries through tOracleInput and tHiveInput. Is there any component that can replace both?
At the moment I have only found tSqoopImportAllTables, but I have been unsuccessful so far...
My goal is to, in someway, reduce the tables loading time. What happens at the moment is that, when i start running my job, it process one "Oracle input" first and after finishing goes to the next "Hive input", consuming time. I was thinking if instead of two "database input" components, I could use one component which could bring the two different databases, saving time.
But you are using two data sources that work in different ways. If you were using two Oracle input components from the same database, it could be done by combining the queries and carrying out the join in the SQL. But since your sources are different, you cannot do this.
How much time is the data loading taking? Maybe there are other ways to improve performance. What is your job built to do?
Depending on what version of Talend Studio (Open Source or Enterprise) you can parallelise the loads from Oracle and Hive to at least acquire the data at the same time.
