topic Re: Extract from FTP server, Transform, and Load into Azure SQL database in Talend Studio

Extract from FTP server, Transform, and Load into Azure SQL database

Anonymous — Sat, 16 Nov 2024 09:44:29 GMT

Hi there,

Here is what we are basically looking for :

1. Import CSV file from FTP or any other server (might be from azure itself)

2. Perform some data quality / preparation operations like cleansing, matching, visualising, etc.

3. Export transformation result directly to Microsoft Azure database table.

Is is possible with any of the available Talend tools, if so which one is that ?

Thanks.

Re: Extract from FTP server, Transform, and Load into Azure SQL database

cterenzi — Tue, 23 May 2017 14:44:43 GMT

Talend has a host of Internet tools which allow you to retrieve files from remote servers. The database components let you extract from or write to a database. Once retrieved, data can be transformed using dozens of components, some specific and some general.

Retrieval and storage:
tFTPGet
tAzureStorageConnection
tAzureStorageGet
tAzureStoragePut
tAzureStorageInputTable
tAzureStorageOutputTable

Transformation:
tMap
tUniqRow
tAggregateRow
tSortRow
tFilterColumns
tFilterRow
tReplace
...

Re: Extract from FTP server, Transform, and Load into Azure SQL database

Anonymous — Tue, 23 May 2017 14:54:42 GMT

Thanks for your reply @cterenzi.

The list of components given by you seem to be helpful. But my main concern is regarding tool. Till now I've just used Talend Data Preparation tool and found there are certain limitations which can not fit for our requirements.

Which tool should I go for, wherein I can use these components :

1. Talend Data Integration, or

2. Data Data Quality, or

3. Other tool

Re: Extract from FTP server, Transform, and Load into Azure SQL database

cterenzi — Tue, 23 May 2017 15:46:14 GMT

I'm only familiar with Open Studio for Data Integration. It can automate a wide variety of tasks, and can handle data validation, matching and cleansing. It's not really a visualization tool, though.

Maybe it's possible to integrate DI with the Data Quality tool? I would ask @xdshi on questions about the product line.

Re: Extract from FTP server, Transform, and Load into Azure SQL database

mks02 — Wed, 24 May 2017 05:04:43 GMT

Talend DI will handle all these tasks/components easily. But confirmation from @xdshi will help better.

Re: Extract from FTP server, Transform, and Load into Azure SQL database

Anonymous — Wed, 24 May 2017 11:20:00 GMT

Thanks
Waiting for confirmation from @xdshi

Re: Extract from FTP server, Transform, and Load into Azure SQL database

Anonymous — Thu, 25 May 2017 03:07:34 GMT

Hello,

From your job requirement description, you need to use both DI(extract data from FTP, tranform and load) and DQ(cleansing, matching, visualising) bundles.

Which talend product are you using? Talend subscription solution for data prep?

Best regards

Sabrina

Re: Extract from FTP server, Transform, and Load into Azure SQL database

Anonymous — Thu, 25 May 2017 07:33:59 GMT

Hello,

I'm done testing our requirements with Talend Data Preparation free desktop. I found its helpful in term of data cleansing, matching, etc., but has certain limitations like row limit = 30K only, can't sort dataset, etc. So didn't give a try to subscription solution of data prep.

Thanks.