topic Replicate jobs with Databricks target has excessive errors in Qlik Replicate

Replicate jobs with Databricks target has excessive errors

broetzel — Mon, 12 Aug 2024 15:51:58 GMT

Specifically for to Databricks jobs and failed to start bulk apply transactions

Hello Qlik Team,

We have several Qlik Replicate jobs that are targeting our Databricks workspace (Unity Catalog) with a source of IBM AS400. These jobs seem to have errors every 1-2 hours each, always with the same message: "Failed to start bulk apply transaction" with additional details explaining "Error while reading file" and "Underlying files have been updated" with the underlying file being specifically one created and managed by Qlik, that we do not otherwise touch. So this seems to be an internal Qlik error, and not something we can fix ourselves.

This error is retriable and sometimes recovers fine, but other times it can cause tasks to silently suspend a table.

It should be noted that we also have several tasks with the same source, but targeting Synapse and those tasks don't seem to have the same problem.

Any guidance would be greatly appreciated.

Re: Replicate jobs with Databricks target has excessive errors

DesmondWOO — Tue, 13 Aug 2024 02:05:23 GMT

Hi @broetzel ,

Thanks for reaching out to the Qlik Community.

I've found a support case that encountered the same error: [DEFAULT_FILE_NOT_FOUND]. It appears that there is a caching issue in Databricks, where requests are made for a file that has already been read.

We have a pre-build patch to address this DEFAULT_FILE_NOT_FOUND error. For more information about this patch, I would recommend creating a support ticket.

Regards,
Desmond

Re: Replicate jobs with Databricks target has excessive errors

john_wang — Tue, 13 Aug 2024 02:43:24 GMT

In additional to @DesmondWOO comment, please have a look at Qlik Article:

Qlik Replicate: DEFAULT_FILE_NOT_FOUND Error

Hope this helps.

John.