Qlik Replicate: Improving Databricks ODBC resilience with EnableRetryWithoutRetryAfterHeader

[shashi_holla]

This article documents how to improve Databricks ODBC resilience with the EnableRetryWithoutRetryAfterHeader parameter. 

Content

• Why does settingEnableRetryWithoutRetryAfterHeader matter?
• The Default Retry Behavior
• Retry behavior withEnableRetryWithoutRetryAfterHeader
• When to apply EnableRetryWithoutRetryAfterHeader
• How to enableEnableRetryWithoutRetryAfterHeader
• Prerequisites
• Configuration
• Errors addressed byEnableRetryWithoutRetryAfterHeader
• HTTP 503 with No Retry-After Header (SqlState: 08S01)
• Delta Transaction Log Truncation (SqlState: 42K03)

The Databricks ODBC driver parameter EnableRetryWithoutRetryAfterHeader is a connection-level setting that controls how the driver responds to server-side failures when the server does not include a Retry-After HTTP header in its response.

By default, this parameter is disabled (0). When enabled (1), it instructs the driver to retry failed requests, even in the absence of explicit retry guidance from the server. This makes pipelines significantly more resilient to transient service disruptions and recoverable error conditions that would otherwise cause tasks to stall indefinitely.

The EnableRetryWithoutRetryAfterHeader parameter is available with driver version 2.9.1+.

Why does setting EnableRetryWithoutRetryAfterHeader matter?

The Default Retry Behavior

The Databricks ODBC driver includes built-in retry logic to handle transient failures. However, by default, this logic only activates when the server includes a Retry-After header in the error response (the standard HTTP signal that tells a client how long to wait before retrying).

In practice, not all Databricks server error responses include this header. When it is absent:

• The driver has no retry signal to act on
• The failure is treated as terminal rather than transient
• The request is abandoned entirely
• Tables dependent on that connection enter a queued state with no further progress

This gap between intent (retry on recoverable errors) and behavior (skip retry without the header) is the source of a class of pipeline failures that can be difficult to diagnose because the driver produces no additional retry activity. It simply stops.

Retry behavior with EnableRetryWithoutRetryAfterHeader

Setting EnableRetryWithoutRetryAfterHeader=1 closes the retry gap. The driver no longer requires the Retry-After header to be present before retrying. Instead, it applies its retry logic to all eligible failures, using a built-in backoff strategy, regardless of whether the server explicitly instructs it to do so.

The practical benefits are:

• Transient 503 errors are recovered automatically: temporary service unavailability no longer causes immediate task failure
• Recoverable Delta read errors are retried: failures that would succeed on a subsequent attempt are allowed to do so
• Queue buildup is prevented: tables are not left stranded in a queued state due to a single unretried failure
• Pipeline stability improves without changes to data architecture or retention policies

When to apply EnableRetryWithoutRetryAfterHeader

This parameter should be considered a standard configuration for any Databricks ODBC connection where pipeline reliability is a priority, particularly in environments that:

• Process high volumes of tables or run long-duration tasks
• Are subject to Delta Lake log retention policies
• Connect to Databricks endpoints that may return 503 responses during maintenance windows or high-load periods
• Have previously experienced unexplained queued states or stalled tasks

How to enable EnableRetryWithoutRetryAfterHeader

Prerequisites

• Databricks ODBC Driver version 2.9.1 or higher (parameter is not available in earlier versions)
• Recommended: upgrade to version 2.9.2 or later for the most stable behavior

Configuration

Add the following to the Databricks Delta endpoint connection using the following internal parameter:

additionalConnectionProperties=EnableRetryWithoutRetryAfterHeader=1

No other changes to the connection string or data architecture are required.

Errors addressed by EnableRetryWithoutRetryAfterHeader

The following errors are direct indicators that the driver encountered a failure without retry-header guidance. Both have been observed to be resolved by enabling the parameter.

HTTP 503 with No Retry-After Header (SqlState: 08S01)

RetCode: SQL_ERROR
SqlState: 08S01
NativeError: 124
Message: [Simba][Hardy] (124) A 503 response was returned but no Retry-After header was provided. Original error: HTTP Response code 503,
TEMPORARILY_UNAVAILABLE: HTTP Response code: 503 [1022502]

Alternative:

RetCode: SQL_ERROR SqlState: 08S01 NativeError: 124 Message: [Simba][Hardy] (124) A 503 response was returned but no Retry-After header was provided. Original error: Unknown

What is happening: The Databricks service returned an HTTP 503 Service Unavailable, a standard, transient condition signaling that the server is temporarily unable to handle the request. The correct behavior is to wait briefly and retry.

Why it fails without the parameter: The driver's retry mechanism is waiting for a Retry-After header that was not included in the response. Without it, the driver takes no retry action and the connection is treated as a hard failure. 

How the parameter helps: With EnableRetryWithoutRetryAfterHeader=1, the driver retries the request using its internal backoff strategy, allowing the pipeline to recover transparently from the temporary service disruption.

Delta Transaction Log Truncation (SqlState: 42K03)

RetCode: SQL_ERROR
SqlState: 42K03
NativeError: 35
Message: [Simba][Hardy] (35) Error from server: error code: '0'error message: 'org.apache.hive.service.cli.HiveSQLException:Error running query: [DELTA_TRUNCATED_TRANSACTION_LOG]
com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException:[DELTA_TRUNCATED_TRANSACTION_LOG]
Unable to reconstruct state at version 16 as the transaction log has been truncated due to manual deletion or the log retention policy (delta.logRetentionDuration=30 days) and checkpoint retention policy
(delta.checkpointRetentionDuration=2 days)

What is happening: Delta Lake could not reconstruct the table state because the transaction log had been truncated, either through manual deletion or by the log retention policy (delta.logRetentionDuration=30 days) and checkpoint retention policy (delta.checkpointRetentionDuration=2 days). The required log file (00000000000000000000.json) was no longer present, preventing reconstruction of the state at version 16.

Why it fails without the parameter: When the driver encounters a server-side error response without a Retry-After header, it does not retry. Conditions that may be transiently recoverable, such as a momentary log unavailability, are never reattempted, and the task stalls.

How the parameter helps: Enabling retries allows the driver to reattempt the read operation, recovering from transient log access issues that do not persist beyond the initial attempt.

Environment

• Qlik Replicate