Pre SQl or Post SQL statements for a table load in Qlik replicate

[choudarybabu]

Can we run Pre SQl or Post SQL statements for a table load in Qlik replicate

We are planning to leverage qlik for all data ingestion needs, currently we cannot use replicate for tables where we dont have primary keys 

We have few cases where we do extract last few months of data everytime and then delete that same months from target table and re insert them , why cant we do this in qlik replicate 

We sometimes just do a full cdc on tables in middleware server, compare source table with target table and then insert new rows and delete changed rows from target , using hashing of rows 

it would be great to have pre sql , post sql support at table level & custom query as source features in replicate
This will eliminate the need to have use ETL tools for data ingestion 

[SwathiPulagam · Accepted solution]

Hi @choudarybabu ,

     We have the Data Movement gateway cloud version to cover all the replicate and ELT use cases in one place. Please reach out to your account representative\sales team for more information.

Thanks,

Swathi

[shashi_holla]

Hi @choudarybabu 

Trying to understand your requirement and my in-line answers for each of the mentioned pain point:

We are planning to leverage qlik for all data ingestion needs, currently we cannot use replicate for tables where we dont have primary keys 

-- Please elaborate, mentioning the source and target DB that's in scope. There can be a workaround for most of the cases.

We have few cases where we do extract last few months of data everytime and then delete that same months from target table and re insert them , why cant we do this in qlik replicate 

-- We can do this in Replicate Full Load process by adding pass thru filters and defining the date conditions.

We sometimes just do a full cdc on tables in middleware server, compare source table with target table and then insert new rows and delete changed rows from target , using hashing of rows 

-- Replicate Full Load with "Do Nothing" mode would help get only the changed records between the source and target.

Please find more details in the Replicate user guide on each of the options suggested.

https://help.qlik.com/en-US/replicate/May2022/Content/Replicate/Main/Introduction/Home.htm

As @SwathiPulagam mentioned the new Cloud "Data Movement gateway" is in the pipeline but it's concentrated more on Cloud Databases and we need to understand your specific requirement.

Thank you,

[choudarybabu]

hello Shashi,

thanks for the reply, I apologize for the late response .

Source is Oracle and destination is Snowflake 

We have used pass thru filters, thats helpful to an extent 

My requirement here but can we use custom queries as source (may be i can workaround this by creating a view with custom sql)

but my question is mostly around a post/pre SQL to be executed on destination db before the load 

can we implement pre/post sql on target db with Qlik replicate 

[shashi_holla · Accepted solution]

Hi @choudarybabu 

--but my question is mostly around a post/pre SQL to be executed on destination db before the load 

Unfortunately No. Replicate cannot but this feature is available in Compose. 

Purpose of Replicate (as the name suggests) is to get your data from source to target. We can add supported transformations but that's the overall scope. You can check if any of the available options for Global or table level transformations would help with your requirement.

Thank you,