Talend in memory?

[Anonymous]

Hi,

i have a question: during ma operation like "sorted", does talend work in memory? This is because i manage a very amaunt of datas (2000000 of record). Thank you! 

[TRF]

Hi,

For some of the components, you have the opportunity to choose to operate in memory or using disk space.

For tSortRow, go to Advanced settings and tick the "Sort on disk" option.

[Anonymous]

But when I use tMap for example and run a substr, does Talend work in memory?

[Anonymous]

1) Open your tMap configuration screen

2) In the top left you have an up and down arrow. Next to that you have "table"- like icon. Click on it

3) The screen that comes up will allow you to handle memory for the tMap and use a temp data directory if you wish

[Anonymous]

I should have added, I am not entirely sure where the example you gave would take place, but my money would be on "in memory". It would seem terribly inefficient to carry out a substr on disk. However I suspect that if the computed value is to be used for further processing (using a tMap variable, etc) that it may use the disk space.

As far as I have considered this, I have always assumed that the disk space option was for storing massive lookups on disk rather than in memory. 

[Anonymous]

I ask this question because, having to handle a great deal of data, I wanted to know where the talenda would perform operations like sorting

[Anonymous]

Because on the basis of where to do such operations I understand what kind of machine I must own, you know?

[Anonymous]

For a Studio machine you want something like the following....

CPU: i7 (or equivalent)
RAM: 16GB (at least)
Disk: An SSD if possible

[Anonymous]

So Talend works mainly in memory, using RAM, but if I want to lean on disk memory, correct?

[Anonymous]

As @TRF said, you can configure many of the components to process the data in memory or on disk. By default most of the processing will take place in memory. FYI I have jobs processing 10s of millions of rows of data running in memory and I usually only assign around 4GB max memory to a job.