topic Difference between tcreatetemporaryfile and Buffer componets in Talend Studio

Difference between tcreatetemporaryfile and Buffer componets

Anonymous — Mon, 10 Sep 2018 07:59:34 GMT

Hi Team

in Talend we have Tcreatetemporaryfile for storing data in temp files and get discarded once they are processed

So is it best practice practice use Tcreatetemporaryfile or buffer components to storing temporary data

please suggest .

Thanks

Saranya

Re: Difference between tcreatetemporaryfile and Buffer componets

fdenis — Mon, 10 Sep 2018 09:11:50 GMT

buffer are to share data between jobs and sub-jobs.(I think there is some trouble managing schemas when you try tu use multiples buffer into the same job. [there is only one buffer by job ] )
Temporary files are real file used to hold data during a process.
You can also use a db Table if you want.

Regards

Re: Difference between tcreatetemporaryfile and Buffer componets

Anonymous — Tue, 11 Sep 2018 14:03:37 GMT

Thank for your replay

ok Leave about Buffers

Lets take Hash Components(Hashinput/HashOutPut). we can temp data in multiple stages of different schemas. we can store and access them.

Here is it better to use hash component or tCreateTemporary File.

Thanks

Saranya

Re: Difference between tcreatetemporaryfile and Buffer componets

Anonymous — Tue, 11 Sep 2018 14:18:31 GMT

Quick answer: it really depends on what you're doing

The hash output/input components will store data in memory. If you're not storing a lot, these can be very convenient and fast. However, you do need to be mindful of memory usage and not load too much data into the hashes.

Using a temp file is almost always preferred for larger amounts of data as the data is written to the filesystem instead of memory. This will be slower, but will allow your jobs to process more data without running out of memory.

Re: Difference between tcreatetemporaryfile and Buffer componets

fdenis — Tue, 11 Sep 2018 14:28:19 GMT

better is temp file to free up some memory.