topic Re: How does DataPrepRun in a Standard job works? in Data Quality

How does DataPrepRun in a Standard job works?

dbeltritti — Sat, 16 Nov 2024 06:43:45 GMT

We've Talend Cloud Data Preparation and I created a recipe based on a dataset that I loaded as test file which is basically doing a regEx replace in a column.

Then I've a standard job that runs on a remote engine onprem that basically reads several files and applies the data preparation recipe using a tDataPrepRun component.

I tested performance using the tDataPrepRun component and a standard regEx tReplace component and the later is like 4 times faster than using the tDataPrepRun.

So I'm wondering how the Data Preparation recipe is applied? Does it downloads the recipe everytime from the cloud and apply it locally in the RunTime engine? or does it upload the data to Talend Cloud, applies the recipe and then downloads it back?

If anyone knows the details please let me know

Thanks!

Re: How does DataPrepRun in a Standard job works?

Anonymous — Wed, 30 Jan 2019 04:25:46 GMT

Hi,

I have raised the query to Product Team using JIRA Ticket and below is the link for your reference.

https://jira.talendforge.org/browse/TDP-6730

Regarding your last query, the data is never loaded to Talend Cloud as Talend Cloud is fetching only metadata information to control the job. All the other details will be processed directly from your remote engine.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: How does DataPrepRun in a Standard job works?

Anonymous — Wed, 30 Jan 2019 08:59:00 GMT

Hi,

When it comes to tDataPrepRun, the following documentation page should answer your question: https://help.talend.com/reader/rGfDn9c_Qjv5~4P5XcYKbw/tClZKcGIQ9tfYAAOSeeg7w. Short version:

For DI jobs, the processing is performed on the Data Prep server
For Big Data jobs, the processing is performed on the cluster

We plan to align the DI behavior to the Big Data one, but there is no confirmed ETA yet. There is no difference between on-prem and Cloud, btw. Same principles apply.

As a side note, the runtime used when running a preparation directly from the UI is described here: https://help.talend.com/reader/94sQcluQTA3Bds1QWGANTw/49m3unRzMnXJnX7tIv79mA

Cheers,

Gwendal

Re: How does DataPrepRun in a Standard job works?

dbeltritti — Wed, 30 Jan 2019 14:25:12 GMT

Thanks for your reply!, just to be sure I got it right

if I use a tDataPrepRun component to use a recipe built on Data Preparation Cloud and the job using that component is being executed in an onPrem remote engine then for every flow of data it will be uploaded to Talend cloud, run the recipe and download the results.

Is that right?

If that's right then it would explain the performance difference with the local regEx

Thanks,

Damian

Re: How does DataPrepRun in a Standard job works?

dbeltritti — Wed, 30 Jan 2019 14:25:39 GMT

Thanks for creating the ticket!, really appreciate it

Re: How does DataPrepRun in a Standard job works?

Anonymous — Wed, 30 Jan 2019 15:54:35 GMT

Yes, that is correct. And yes, that fully explains the performance discrepancy ... and why we want to review the way DI jobs work with tDataPrepRun to mimic Big Data jobs.

Regards,

Gwendal