topic Re: What component to use to store temporary data before it is then transfer to the final destination database in Talend Studio

What component to use to store temporary data before it is then transfer to the final destination database

Anonymous — Sat, 16 Nov 2024 03:03:49 GMT

I want to create a job with the following sequence

Source DB ----- > Job to filter the data and store in temporary (virtual) table ----> Job to query data from temporary (virtual) table ----> job to transfer data to the final destination DB table. The flow basically illustrated below.

FYI i know how to directly transfer from source to the destination...but the query is too heavy...that's why i need to store in temporary table with filtered data first before finally transfer it to the final destination db. How it can be done please enlighten me

Re: What component to use to store temporary data before it is then transfer to the final destination database

manodwhb — Fri, 06 Mar 2020 10:04:19 GMT

@talendet , if you want t store in memory you can use tHasOutput component to store data in between the flow and you can read same using tHashInput.

check the below link to know more about those components.

https://help.talend.com/reader/8_lXEInq2ehdXds6XE9PsA/kQWU35YXytKECckZ5EqvGg

Re: What component to use to store temporary data before it is then transfer to the final destination database

Anonymous — Fri, 06 Mar 2020 10:06:52 GMT

Can be done using MYSQL? and it seems tHashOuput component not available in the 5.4.1 version

Re: What component to use to store temporary data before it is then transfer to the final destination database

manodwhb — Fri, 06 Mar 2020 10:11:43 GMT

@talendet, it is a component what it will connected from source component that data will be stored in memory and you read it.

check the below link to know more about it

https://help.talend.com/reader/8_lXEInq2ehdXds6XE9PsA/kQWU35YXytKECckZ5EqvGg

Re: What component to use to store temporary data before it is then transfer to the final destination database

Anonymous — Fri, 06 Mar 2020 10:32:14 GMT

How do i perform mysql select statement from the thashinput buffer ? do i need to create another tMysql input after it? see this is the incomplete flow which i dont know how to continue with it. If it can be queried using mysql select from what? the same table name or ?

Re: What component to use to store temporary data before it is then transfer to the final destination database

manodwhb — Fri, 06 Mar 2020 10:45:05 GMT

@talendet , you cannot select that way,what ever your are loading into tHasoutput that data will be read in tHasInput.

Re: What component to use to store temporary data before it is then transfer to the final destination database

manodwhb — Fri, 06 Mar 2020 10:45:55 GMT

@talendet , Why you want intermediate virtual table?

Re: What component to use to store temporary data before it is then transfer to the final destination database

Anonymous — Fri, 06 Mar 2020 16:52:39 GMT

hello,

you can use tHashInput more than once in the job, or tBufferInput only once in the job.

Re: What component to use to store temporary data before it is then transfer to the final destination database

Anonymous — Sat, 07 Mar 2020 13:05:54 GMT

If you are asking why i want intermediate virtual table is because i don't want the calculation job to query direct from the source database which is very heavy. The source database is actively being used by another application where users entering their data. So to prevent the heavy load on the source database i need the intermediate virtual table so the calculation will be query from the intermediate virtual table instead of directly on the DB.

By anyway i have solved the problem differently which provide the exact solution i wanted. The job goes like this

Source DB ---> Transfer to Destination DB table A ----> Query from Destination DB Table A ----> Transfer to Destination DB Table B ----> Delete Destination DB Table A.

Re: What component to use to store temporary data before it is then transfer to the final destination database

Anonymous — Tue, 14 Apr 2020 01:23:13 GMT

Hi @talendet , can you please post a screenshot of your job? I try to do the same thing with the temporary checkbox checked in MySQL 5, but even tho, I run this in a seperate subjob, I constantly get an error saying that my temp table does not exists: