Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
I'm trying to replicate data from source (Oracle) to target (S3). While replicating, I'd like to make the 'Change table' name/location based upon required file creation time like below FORMAT:
TABLENAME/YEAR/MONTH/DAY/HOUR
with YEAR/MONTH/DAY/HOUR coming from replication task runtime.
I'm able to create required S3 location the first-time tasks run for Change table using the Global Rules transformation with expression:
Transformation Rule - Rename Change Table with expression:
'/'||$AR_M_SOURCE_TABLE_NAME||'/'||strftime('%Y',DATE())||'/'||strftime('%m',DATE())||'/'||strftime('%d',DATE())||'/'||strftime('%H',TIME())
example -- TABLENAME/2025/04/22/14
which is successfully being created the first time, but as soon as the time (date or hour) changes for incremental updates from source, error is generated: for the new folder path
"Getting DB object (''S3'.'dbo'.'/TABLENAME/2025/04/23/09'') info from repository failed.
Error finding object: 'S3'.'/TABLENAME/2025/04/23/09' in repository"
---------
In short, is there a way to have different Change tables based on change files creation time?
Or we can't change the target change table name/location created in the initial task run?
Just want to be sure, can we achieve this functionality through Qlik Replicate?
Hello @TNJ ,
Thanks for reaching out!
The expression works for me, both the Full Load and CDC stages generated correct file path and files, see results of task "Ora-to-S3":
Full Load stage:
CDC stage:
Please take note that, $AR_M_SOURCE_TABLE_NAME must be preceded by a space and followed by a space.
So the complete expression of the Global Transformation rule is:
Rename table for %.% to '/'|| $AR_M_SOURCE_TABLE_NAME ||'/'||strftime('%Y',DATE())||'/'||strftime('%m',DATE())||'/'||strftime('%d',DATE())||'/'||strftime('%H',TIME())
Hope this helps.
John.
Thanks @john_wang.
The expression worked for me as well in the initial run.
BUT
Will a new folder be created for next hour run?
LIKE--
CDC stages -- KIT/2025/04/23/21_ct
And for next day run?
CDC stages -- KIT/2025/04/24/11_ct
otherwise, does Change table has to be same for all future runs?
Because as per requirement, there is need to run the task at a 6-hour interval. so will be stopping and resuming the task and will have different hour value.
Hello @TNJ ,
Would you please open support ticket? Our support team will be more than happy to assist you.
Thanks,
John.
Thanks @john_wang, have raised the ticket and got an update.
The change table once created can't be changed or updated/renamed.
Hello @TNJ ,
That behavior is expected, as the table name was generated once during task startup. What happens if you stop and then resume the task? We may need to find a way to force Replicate to regenerate the table path and name.
Regards,
John.