Skip to main content
Announcements
UPGRADE ADVISORY for Qlik Replicate 2024.5: Read More
cancel
Showing results for 
Search instead for 
Did you mean: 
david_lange
Contributor II
Contributor II

Full load to temp with rename

I would like the QLIK replicate full load to be transparent to end-users querying the table.

Can I reload to a temp table and rename it to minimize any disruption to end-user queries?

How can this be accomplished within the tool?

The full load is required.

The source object is an Oracle VIEW (with parallel hint) over a materialized view snapshot table.

The target object is a Snowflake table.

-Dave

Labels (3)
1 Solution

Accepted Solutions
Steve_Nguyen
Support
Support

@david_lange ,, out of the box Replicate can not automate the process.

 

however, you can use QEM API to start the task full load, check on task status when complete and have your script do the two options after full load is complete .

Help users find answers! Don't forget to mark a solution that worked for you! If already marked, give it a thumbs up!

View solution in original post

5 Replies
Heinvandenheuvel
Specialist III
Specialist III

[Edit: meant to ask - is the full-load only or CDC as well. CDC makes rename even harder if not impossible]

 

I understand the desire to load to temp tables and switch when done.

This is not a new request and many years ago an attempt was made to implement this for SQLserver target based on a specific customer request.  I suspect the code is still there controlled by an advance, internal, parameter  loadToTempTables.

The basic required actions are clear: on successful table full-load,  drop the original table, make sure the temp table protections are as they should be for the real table, rename.

The error handling and timing challenges are less clear, similar to a direct full load but just less clear. While it worked, the customer at that time decided NOT to use it in production and with that the availability remained limited to (5) SQLserver family targets and was never 'advertised', only visible by figuring out how to see internal parameters, guess at their usage and verify with reptasks and target db SQL LOGS.

You could (should?) consider an feature request. Why not. Just don't hold your breath while waiting.

Instead figure out how to use a Replicate API (preferably the Enterprise Manager, but REPCTL command will work also and make yourself a script looking at "full_load_completed","full_load_start","full_load_end" (and possibly  full_load_counters) from GetTaskDetails or more likely GetTableStatuses looking for per-table "table_full_load_info" and status. A good bit of work, but only done every minute or so and computers don't mind.

Or one could scan the reptask log(s) for table load completed messages every so often remembering where one left off and mindful of logs being rolled.

Details you depend on whether managing a full load for all tables, for some, and whether some table have foreign keys and are best renamed when all all done (something Replicate could/would not know).

Hope this helps some!

Hein.

 

 

shashi_holla
Support
Support

I'm assuming that you already have the Target table in place with the current set of data and in need to do a Full Load without truncating or recreating the existing table.

I would accomplish this by changing the target table to <table_name>_test in the Replicate table settings and run the Full Load:

shashi_holla_0-1681166295586.png

And once the Full Load is complete, have two options:

1) Drop the original table and rename _test table to original name

or

2) Sync the data from _test to the original table and drop the _test table.

 

Thank you,

 

david_lange
Contributor II
Contributor II
Author

Can these steps be automated via the QLIK replicate tool?


"And once the Full Load is complete, have two options:

1) Drop the original table and rename _test table to original name

or

2) Sync the data from _test to the original table and drop the _test table."

Steve_Nguyen
Support
Support

@david_lange ,, out of the box Replicate can not automate the process.

 

however, you can use QEM API to start the task full load, check on task status when complete and have your script do the two options after full load is complete .

Help users find answers! Don't forget to mark a solution that worked for you! If already marked, give it a thumbs up!
john_wang
Support
Support

Hello @david_lange ,

Agree with @Heinvandenheuvel , @Steve_Nguyen  and @shashi_holla .

Not sure if your source table/view contains PK or Unique Index, or the view has a de facto 'logical' PK (for example we can make a column as PK in target table). If yes then we may make it automated; Otherwise we've no chance to trigger the 'renaming' operation. Let us know and we will confirm for you.

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!