topic Re: Talend in memory? in Talend Studio

Talend in memory?

Anonymous — Sat, 16 Nov 2024 09:37:35 GMT

Hi,

i have a question: during ma operation like "sorted", does talend work in memory? This is because i manage a very amaunt of datas (2000000 of record). Thank you!

Re: Talend in memory?

TRF — Thu, 15 Jun 2017 09:09:04 GMT

Hi,

For some of the components, you have the opportunity to choose to operate in memory or using disk space.

For tSortRow, go to Advanced settings and tick the "Sort on disk" option.

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 09:29:17 GMT

But when I use tMap for example and run a substr, does Talend work in memory?

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 09:48:59 GMT

1) Open your tMap configuration screen

2) In the top left you have an up and down arrow. Next to that you have "table"- like icon. Click on it

3) The screen that comes up will allow you to handle memory for the tMap and use a temp data directory if you wish

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 09:52:28 GMT

I should have added, I am not entirely sure where the example you gave would take place, but my money would be on "in memory". It would seem terribly inefficient to carry out a substr on disk. However I suspect that if the computed value is to be used for further processing (using a tMap variable, etc) that it may use the disk space.

As far as I have considered this, I have always assumed that the disk space option was for storing massive lookups on disk rather than in memory.

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 09:59:41 GMT

I ask this question because, having to handle a great deal of data, I wanted to know where the talenda would perform operations like sorting

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 10:08:34 GMT

Because on the basis of where to do such operations I understand what kind of machine I must own, you know?

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 10:23:02 GMT

For a Studio machine you want something like the following....

CPU: i7 (or equivalent)
RAM: 16GB (at least)
Disk: An SSD if possible

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 10:25:53 GMT

So Talend works mainly in memory, using RAM, but if I want to lean on disk memory, correct?

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 11:08:33 GMT

As @TRF said, you can configure many of the components to process the data in memory or on disk. By default most of the processing will take place in memory. FYI I have jobs processing 10s of millions of rows of data running in memory and I usually only assign around 4GB max memory to a job.

Re: Talend in memory?

Anonymous — Thu, 15 Jun 2017 11:12:10 GMT

Ok, thanks for all!! Can I ask you if you can help me with another issue posted in the post: Problem timestamp tOracleOutputBulkExec

Re: Talend in memory?

Anonymous — Fri, 21 Jul 2017 05:04:17 GMT

Hi,

@livio.ventura wrote:
Ok, thanks for all!! Can I ask you if you can help me with another issue posted in the post: Problem timestamp tOracleOutputBulkExec

Please give us more information about your problem timestamp tOralceOutputBulkExec or create a new topic on forum for your issue.

Best regards

Sabrina