I want to load a csv file into RDBMS. But, I want to design a job such that if loading fails in between, then it should start loading from where it failed. How can we design this job?
Assuming you're loading 1 CSV row = 1 RDBMS row, then you can achieve this, but does require some careful planning with the columns in your RDBMS. 1st of all, you'll need to have a column that represents the filename the source data came from, ensuring that each source CSV is a unique file name.
When you're given a file to load (or a failure from a previous execution), you count how many rows in the RDBMS have been loaded with that filename., so you might get back zero, some or all rows. When your job is running, use a tFilterRow to only allow rows where their row number (Numer.sequence) is greater than those loaded in the RDBMS. This also stops you reloading the same file twice.
I've used a pattern like this when a DB connection we were using was a little flaky
Thanks
David
If these answers are useful, don't forget to give Kudos
