File Sources: How do we move the files after processing them?

[badgerriaan]

Hello, we have some tasks that read delimited text files as sources and put them into MS SQL target.

Is there a way to move the files to a different location(Folder) after they have been processed successfully?

E.g. The files are dropped in F:\Share\Source and then after processing must be moved to F:\Share\Processed

I have tried to set the values that I found on the Endpoint Connection itself, Under Change Processing Cleanup, but this doesn't seem to want to work for me. Nothing happens.

We are using: Qlik Replicate May 2021 (2021.5.0.1272), running on Windows Server 2019 in Azure

Qlik Replicate 

[Heinvandenheuvel]

The picture is handy but does not show enough.

For example, is this a full load task (single shot) or CDC taks which is continuously running? Using reference files? The post-processing  options are only for CDC tasks. (UPDATE:  I've been informed it is probably also valid for fullload)

Please export the task and attach the json - obfuscating sensitive information as needed.

You may also want to attach a task logs, probably after setting source_capture to trace or even verbose. Try it first, and study the log first yourself ofcourse trying to understand the various steps Replicate undertakes.

Good luck,

Hein

[Heinvandenheuvel]

@badgerriaan , any updates? Did you figure it out? 

I wrote that the UI cleanup choices might would only work for CDC tasks, inspired by the UI indicating "Change Processing Cleanup" and the Userguide had similar wording. I've been informed it is probably also valid for fullload if that's relevant to you use case.

Hein.

[shashi_holla]

One other idea if the Replicate cannot handle automatically then we can create a separate .bat task scheduler job in windows server to perform this operation of moving the file once a week or more frequently as required.

[gvkiranreddy325]

Even i am looking for the same option availability in Qlik Replicate as it is having same kind of limitations here.

Please let me know if you find any options for the same?

Else we have to find an option for processing last updated files from the last run.

Say suppose, we have total ten files on 2022-02-03 in F:Share/ Processed/

and they were processed successfully on 2022-02-03 and remain they are available in same location.

Now again we have got another ten files in same location on next day 2022-02-04, so we would be having total 20 files. So, Qlik replicate has not to pick up total 20 files instead, it should pickup only that last updated files from the last run, that means, it should run only 10 files from F:Share/ Processed/ on 2022-02-04.

Do you any option for this at least?

[Steve_Nguyen · Accepted solution]

if you are using file as source, 

there is option for :

Change Processing Cleanup

Select one of the following cleanup options to determine what Replicate should do with the processed Change Files/Reference Files:

Archive files to folder - to archive the file(s) to the specified location.

---

user guide:

https://help.qlik.com/en-US/replicate/November2022/Content/Replicate/Main/File/Advanced_Settings.htm

[gvkiranreddy325]

I have file as a source, my goal is to process the delimited text files into Snowflake target database from FTP location.

We used to receive the file from source vendor for every one hour, files used to append in inbound location.

If we have any facility to process the very latest files with Qlik replicate, then we don't need to archive them, even after process completion.

If don't have option for latest file processing in Qlik Replicate then i would need to look for the option in Qlik Replicate to archive the files to other location.

I have tried the options that you specified, but they are not getting archived.

Can you help me with little detailed steps?

Thanks in advance.

[john_wang]

Agree with Hein, the "Change Processing Cleanup" options apply to "Change Files" and/or "Reference Files" only, it does not work for Full Load data files. So far the workaround to cleanup the Full Load data files:

1. "Stopping the Task after Full Load" done

2. Call a cleanup job command scripts job manually

hope this helps.

Regards,

John.

[gvkiranreddy325]

Do we have an option in Qlik Replicate to pickup the latest files for processing from the file location when it process everytime by avoiding the old files? So that we don't need to bother about this archiving files? Please suggest.

Hence this option is not enable in my Qlik Replicate login.