topic Re: load time databricks in App Development

load time databricks

jjustingkm — Thu, 08 May 2025 01:52:24 GMT

I am trying to load a big table from databricks around 100 million records and is taking around 1.5 hours.

Needed the detail data in the sheet.

if we load the data into a QVD, would it make any difference in the load time ( I know that loading into a table and QVD both needs to hit the database).

Re: load time databricks

paulcalvet — Thu, 08 May 2025 06:53:17 GMT

Hello,

Yes the best solution in your case is to load from databricks in delta mode and store data into QVD.

You can split QVD.

Load from QVD is faster in Qlik and your records will be load in few minutes.

Re: load time databricks

jjustingkm — Thu, 08 May 2025 18:59:38 GMT

Thank you for your reply. I am basically looking for reduce the load from databricks.

When you say delta mode, are you talking about incremental load. In this case there is no modification date in the table so incremental load is not possible.