Best possible ways to read Delta format Files and insert into a synapse table
Hello there,
My objective is to read data from Azure Delta (tables/files) and INSERT the date into a Azure synapse table through Talend (using Dynamic Schema). I'm trying to explore all the possible options available.
I've tried multiple options but had different issues.
a. Tried to read data using DeltaLake component (jdbc) and INSERT data using synapseOutput component.
synapseOutput component doesn't support dynamic datatype and if I select standard schema, the INSERTS are really slow (5 rows/sec)
Tried to use synapseBulk, which also doesn't support dynamic.
b. Tried to read data using Deltalake component (jdbc) and INSERT data using tDBOutput (Synapse via jdbc)
tDBOutput
(Synapse via jdbc)
component INSERT speeds are really slow (same a synapse component - 5 rows/sec)
Tried using BULK components, they are failing with weird errors!! (for which I have raise a different topic)
c. Current temporary working approach ; Reading
data using Deltalake component (jdbc) and write a csv file on gen2 storage
account, using COPY INTO read that csv back and INSERT into synapse table.
Though option C is working, it isn't viable as option A and B.
I would like to know if there any other options available, which I may have overlooked.
