Failed to write a file to the final destination

[toni_lajsner]

Hi all

Sometimes we are getting following log messages :

Failed to move data file 'F:\Attunity\Replicate\data\tasks\file.csv' to the final destination.

and 

Failed to check if a file should be closed or failed to close a file.

and

Failed to write a file to the final destination.
Write entire file failed: source = '.csv' open type = 2
Failed to write entire file (second trial)
Failed to upload <.csv>
JAVA_EXCEPTION, message: 'io.swagger.client.ApiException: java.net.UnknownHostException: depprdweuadls002.dfs.core.windows.net'

My question is what is the best practice as we do not like reload the tables as they can be large.

We are using ADLS as our target with following File size 20480 kb

[Pedro_Lopez]

Hi Toni,

Thanks for reaching out. Usually these errors are related to a disconnect from the source. Do you see any other errors coming in the logs? Do you know if the Source got disconnected at some point?

Regards,

Pedro

[toni_lajsner]

Do we need to be concerned about the massage , I am thinking about missing a records and reloading the task?

[Pedro_Lopez]

Hi Toni,

This message could lead to data missing as the file is not closed correctly. Did you have any disconnect from source? 

I'd suggest raising a support case with us, enable TARGET_APPLY on TRACE and attach the diagnostic package of the task to the case for further analysis.

Regards,

Pedro 

[john_wang]

Hello @toni_lajsner ,

In general these errors/warnings will not cause data lost, as Replicate will retry to transfer the interim file again or reload the whole table again automatically. You may confirm that by reading the task log file, if you see below line after the error messages lines:

2023-01-13T05:05:11 [TASK_MANAGER ]I: Reloading table 1 because subtask #1 finished with error (replicationtask.c:2644)

And then you may probably see other lines :

2023-01-13T05:39:42 [TASK_MANAGER ]I: Loading finished for table 'xxx'.'yyy' (Id = 1) by subtask 1. <nnnnn> records transferred. (replicationtask.c:3012)

where xxx is schema name, yyy is table name, nnnnn is the rows number.

If you can see above lines then no worries, all rows will be transferred to target side by the seconds retry in the Full Load stage.

Hops this helps.

Regards,

John.

[toni_lajsner]

I am not seeing "TAST_MANAGER" reload the table or loading is finished, only that connection to our target failed and “failed while preparing stream component”. 

Also looking at a massage: Target last committed record id from the previous run is '***' (streamcomponent.c:1664) i cannot find this record id anywhere, is that Qlik’s record id and how can I use this information to see if record is added to our target?

[john_wang]

Hello @toni_lajsner ,

There are 3 major types of information in Replicate task log file, they are ERROR, WARNING, and INFORMATION catalogs, from task log file we may see the components name and the logging type, for example "]E:", "]W:", & "]I:" correspondingly. 

For your error, "failed while preparing stream component", in general it looks like "[TARGET_LOAD ]E:", that means the target endpoint connection cannot be established, or the target component cannot startup. For this type error, you need not set additional logging level, it will print always upon an error.

Not sure it's random error, or persist. We'd like suggest you set components TARGET_LOAD & TARGET_APPLY to Verbose and check if it helps to solve the problem, if you need support team help, please open a ticket and attach the task Diagnostics Packages, we'd love to help.

Regards,

John.