topic How to read csv file from where it failed to load in Talend Studio

How to read csv file from where it failed to load

firdos4u — Sat, 16 Nov 2024 00:46:57 GMT

I want to load a csv file into RDBMS. But, I want to design a job such that if loading fails in between, then it should start loading from where it failed. How can we design this job?

Re: How to read csv file from where it failed to load

manodwhb — Sun, 24 Jan 2021 04:45:00 GMT

@Not defined Not defined ,check the below link is useful.

https://help.talend.com/r/pzEsR9WAQlQkz4tFLp6TpA/omFXPdTqrImC7U0HX5iVVQ

Thanks,

Manohar

Re: How to read csv file from where it failed to load

David_Beaty — Mon, 01 Feb 2021 17:40:12 GMT

Hi,

Assuming you're loading 1 CSV row = 1 RDBMS row, then you can achieve this, but does require some careful planning with the columns in your RDBMS. 1st of all, you'll need to have a column that represents the filename the source data came from, ensuring that each source CSV is a unique file name.

When you're given a file to load (or a failure from a previous execution), you count how many rows in the RDBMS have been loaded with that filename., so you might get back zero, some or all rows. When your job is running, use a tFilterRow to only allow rows where their row number (Numer.sequence) is greater than those loaded in the RDBMS. This also stops you reloading the same file twice.

I've used a pattern like this when a DB connection we were using was a little flaky

Thanks

David

If these answers are useful, don't forget to give Kudos