Oracle to ADLS Full load is failing with ]E: Failed to convert file from csv to parquet

[eksmirnova]

Hello,

We have a Full load + Store changes task which loads data from Oracle to ADLS(parquet snappy format). But it is failing for one table during Full load with the error:

00018089: 2024-06-05T05:47:20 [TARGET_LOAD ]E: Failed to convert file from csv to parquet [1024902] (file_utils.c:899)

00018089: 2024-06-05T05:47:20 [TARGET_LOAD ]E: Failed to convert file '/data/replicate/qlik/tasks/task_name/data_files/MY_TABLE_NAME/LOAD0000000E.tmpcsv'. [1024902] (file_imp.c:2538)

00018088: 2024-06-05T05:47:20 [SOURCE_UNLOAD ]I: Unload finished for table 'FS'.'MY_TABLE_NAME' (Id = 6). 16436666 rows sent. (streamcomponent.c:3784)

00018033: 2024-06-05T05:47:20 [TASK_MANAGER ]W: Table 'FS'.'MY_TABLE_NAME' (subtask 1 thread 1) is suspended. (replicationtask.c:3147)

Any suggestions to resolve that? 

[john_wang]

Hello @vinayak_m ,

Various reasons may lead the same error for example an invalid DATE value etc. Let's try to troubleshoot the issue by:

1- Set Logging component TARGET_LOAD to Verbose

2- Reduce the maximum file size (just for troubleshooting purpose, the CSV file is easy to read, If you know the 'problematic' rows PK values then it's better to add it the filter to minimaze the temp CSV files)

3- Add internal parameter in target S3 endpoint: keepCSVFiles set to TRUE

4- Recreate the issue then check the task log file and the temp CSV files to understand the root cause. 

Hope this helps.

John.