Job hangs while processing insert in databricks table

[Jacco_De_Zwart]

Hi all,

I have a simple job :

tdbinput --> tmap --> tdboutput

The input is an MS-SQL server table with 67.819 rows. All I want is to move these to a databricks table.

All is setup with connections.

Problem is, no matter what I do, the job 'hangs' after inserting all records.

I have already increased the -Xmx JVM argument to -Xmx4g, tried different batch and 'commit every' sizes (ranging from 100 to 100.000). 

There is no commit component for dataBricks. The databricks connection component has auto-commit on, but that doesn't help.

So, I'm out of options.

Can anyone help me out ? This should be a basic job, just move records from one source to the other.

Thanks in advance.

Jacco

[Jacco_De_Zwart]

Small update . . . . the commit seems to be working but is very slow on the databricks side.