Qlik Replicate and Google Cloud Storage: Failed to convert file from csv to parquet

[john_wang]

When using Google Cloud Storage as a target in Qlik Replicate, and the target File Format is set to Parquet, an error may occur if the incoming data contains invalid values.

This happens because the Parquet writer validates data during the CSV-to-Parquet conversion. A typical error looks like:

[TARGET_LOAD     ]E:  Failed to convert file from csv to parquet
Error:: failed to read csv temp file
Error:: std::exception [1024902]  (file_utils.c:899)

Environment

• Qlik Replicate all versions
• Google Cloud Storage all versions

Resolution

There are two possible solutions:

• Clean up or remove the incorrect records in the source databases
• Or add a transformation to correct or replace invalid dates before they reach the target.

Cause

In this case, the source is SAP Oracle, and a few rare rows contained invalid date values. Example: 2023-11-31.

By enabling the internal parameters keepCSVFiles and keepErrorFiles in the target endpoint (both set to TRUE), you can inspect the generated CSV files to identify which rows contain invalid data.

Internal Investigation ID(s)

00417320