topic Re: Redshift post-processing in Qlik Replicate

Redshift post-processing

simonB2020 — Sat, 30 Apr 2022 22:25:36 GMT

I need to execute some post processing script on my Redshift target once the full load has completed.

i.e. call a stored procedure

Is this possible to trigger from Replicate

Thanks

Re: Redshift post-processing

SwathiPulagam — Sat, 30 Apr 2022 22:56:38 GMT

Hi @simonB2020 ,

For the Redshift target, there is no option to run a post-processing script or stored procedures from Qlik Replicate.

Thanks,

Swathi

Re: Redshift post-processing

john_wang — Sun, 01 May 2022 02:43:09 GMT

Hello @simonB2020 , copy @SwathiPulagam ,

Maybe you can STOP the task automatically after the Full Load completion by enabling option "Before cached changes have been applied". Then run your custom scripts and RESUME the task again after the scripts execution.

Hope this helps.

Regards,

John.

Re: Redshift post-processing

Heinvandenheuvel — Mon, 02 May 2022 17:57:39 GMT

Replicate Build-in post processing command are limited to CSV type file processing.

I think your best bet is to create a polling script either using the REPCTL commandline gettaskstatus on the repiicate server, or the Enterprise Manager gettaskstatus Api call from anywhere to find the current task status.

You would look for state =STOPPED

When state=running, just wait and check again. Optionally report current time, full_load_completed_tables_count / full_load_loading_tables_count / full_load_queued_tables_count

When state=error - report via Email or similar optionally indicating start_time, and various table counts

When state=stopped verify you further details for 'reasonable values' to decide to call it good and run post-processing of report failure after all.

full_load_completed=1
full_load_finish_time > last recorded starting time or 'yesterday' or whatever is appropriate.
full_load_completed_tables_count = expected count
full_load_total_record_transferred >= some expected minimum (yesterday's number?)
full_load_error_tables_count = 0

Hein