What component to use to store temporary data before it is then transfer to the final destination database

[Anonymous]

I want to create a job with the following sequence 

Source DB ----- > Job to filter the data and store in temporary (virtual) table  ----> Job to query data from temporary (virtual) table ----> job to transfer data to the final destination DB table. The flow basically illustrated below.

FYI i know how to directly transfer from source to the destination...but the query is too heavy...that's why i need to store in temporary table with filtered data first before finally transfer it to the final destination db. How it can be done please enlighten me

[manodwhb]

@talendet , if you want t store in memory you can use tHasOutput component to store data in between the flow and you can read same using tHashInput.

check the below link to know more about those components.

https://help.talend.com/reader/8_lXEInq2ehdXds6XE9PsA/kQWU35YXytKECckZ5EqvGg

[Anonymous]

Can be done using MYSQL? and it seems tHashOuput component not available in the 5.4.1 version

[manodwhb]

@talendet, it is a component what it will connected from source component that data will be stored in memory and you read it.

check the below link to know more about it

https://help.talend.com/reader/8_lXEInq2ehdXds6XE9PsA/kQWU35YXytKECckZ5EqvGg

[Anonymous]

How do i perform mysql select statement from the thashinput buffer ? do i need to create another tMysql input after it? see this is the incomplete flow which i dont know how to continue with it. If it can be queried using mysql select from what? the same table name or ?

[manodwhb]

@talendet , you cannot select that way,what ever your are loading into tHasoutput that data will be read in tHasInput.

[manodwhb]

@talendet , Why you want intermediate virtual table?

[Anonymous]

hello,

you can use tHashInput   more than once in the job, or tBufferInput   only  once in the job.

 

[Anonymous · Accepted solution]

If you are asking why i want intermediate virtual table is because i don't want the calculation job to query direct from the source database which is very heavy. The source database is actively being used by another application where users entering their data. So to prevent the heavy load on the source database i need the intermediate virtual table so the calculation will be query from the intermediate virtual table instead of directly on the DB.

By anyway i have solved the problem differently which provide the exact solution i wanted. The job goes like this 

Source DB ---> Transfer to Destination DB table A ----> Query from Destination DB Table A ----> Transfer to Destination DB Table B ----> Delete Destination DB Table A. 

[Anonymous]

Hi @talendet , can you please post a screenshot of your job? I try to do the same thing with the temporary checkbox checked in MySQL 5, but even tho, I run this in a seperate subjob, I constantly get an error saying that my temp table does not exists: