Best possible ways to read Delta format Files and insert into a synapse table

vbadri — Sat, 16 Nov 2024 00:07:42 GMT

Hello there,

My objective is to read data from Azure Delta (tables/files) and INSERT the date into a Azure synapse table through Talend (using Dynamic Schema). I'm trying to explore all the possible options available.

I've tried multiple options but had different issues.

a. Tried to read data using DeltaLake component (jdbc) and INSERT data using synapseOutput component.

synapseOutput component doesn't support dynamic datatype and if I select standard schema, the INSERTS are really slow (5 rows/sec)

Tried to use synapseBulk, which also doesn't support dynamic.

b. Tried to read data using Deltalake component (jdbc) and INSERT data using tDBOutput (Synapse via jdbc)

tDBOutput

(Synapse via jdbc)

component INSERT speeds are really slow (same a synapse component - 5 rows/sec)

Tried using BULK components, they are failing with weird errors!! (for which I have raise a different topic)

c. Current temporary working approach ; Reading

data using Deltalake component (jdbc) and write a csv file on gen2 storage

account, using COPY INTO read that csv back and INSERT into synapse table.

Though option C is working, it isn't viable as option A and B.

I would like to know if there any other options available, which I may have overlooked.

Tickets I've raised :

Issue with tAzureSynapseOutput and tDBOutput (talend.com)

Issue with tDBOutputBulk, tDBBulkExec & tDBOutputBulkExec (talend.com)

Kind Regards

Re: Best possible ways to read Delta format Files and insert into a synapse table

Anonymous — Wed, 27 Apr 2022 03:02:03 GMT

Hello,

When the customer wants to load large, we suggest they use tAzureSynapseBulkExec component.

It supports 2 ways to load large data into Azure synapse:

Best regards

Sabrina

topic Best possible ways to read Delta format Files and insert into a synapse table in Talend Studio

Best possible ways to read Delta format Files and insert into a synapse table

Re: Best possible ways to read Delta format Files and insert into a synapse table