Just looking for any guidance or suggestions on how to tackle a problem we have at my workplace.
We have an old data warehouse where 15 years of old SSIS jobs populate it with a lot of technical debt and knowledge of how they work no longer with the company.
We have a new data warehouse spun up for an ERP migration project and some of the data that gets populated with SSIS is required on the new data warehouse as a short term stop gap for business reporting continuity.
This stop gap will be cleaned up post go live of the ERP project but for now we need a quick and efficient way to replicate the SSIS populated table data from old data warehouse to new data warehouse.
The old data warehouse is MSSQL 2014 Standard and does not have CDC as a result...
I've created python code that replicates the data using pandas and it works reasonably well other than the length it takes to replicate and the amount of RAM the pandas dataframe consumes. So i can only run this once a day meaning some data will be a bit stale in the new data warehouse till it runs again the next day.
I'm trying to come up with a way to be able to run this replication more often given time and ram limitations.
One idea I had was to hash the table contents from the last run and save it, then check the hash to see if any of the table changed before using valuable replication time/ram resources. Has anyone ever done anything like this ? Any libraries that can help me ?
Or any other completely different ways to handle this without CDC or large amounts of hardware spend ?
Hi thank you for the post to the QDI Forums. For this issue are you using the old AIS Connect product for this old environment using SSIS? Please confirm as per this product has been marked as End of Life. Please confirm along with screen shots of the older Solution on this older AIS Product.
