Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hey Everyone,
I am loading in multiple CSVs and there has been a new column added to the new CSVs. I've noticed it will not load in the old ones that are missing the new column. Is there a way to ignore the error and still load in all the CSVs and just put nulls under the columns for the older CSVs?
Could you put some example or your code please?
supposing you keep all your csvs, and that you have identified the date (datefield) when this column was added;
your_table:
Load
a,
b,
'' as c, //your column storing null when datefield is inf than change date
datefield
from X where datefield < '01/08/2018'; //date from wich the column was added
concatenate
Load
a,
b,
c, //your column
datefield
from X where datefield >= '01/08/2018'; //date from wich the column was added
A specified field must always exists - otherwise the load will fail with an error (errors could in general be skipped but in this case it won't be useful because the data won't be loaded).
The easiest way to handle it will be to move the new csv's into another folder and just using two different load-statements whereby the second load-statement get a concatenate(FirstTable) prefix.
Also quite easy is it to use a dummy-load and concatenate your csv there with something like:
FirstTable: load 'dummy' as Dummy autogenerate 0;
concatenate(FirstTable) load * from csv;
drop field Dummy; // maybe further fields if you don't need all from the csv
Also possible but with more efforts is to load the files with a filelist-loop and checking there within an in front load if this field exists to brach in different load-statements and/or reading all existing fields in variables and creating with them a load-statement on the fly.
- Marcus
You can set Errormode in the script.
set ErrorMode=0;
It will simply ignore the failure and continue script execution at the next script statement.
More details are available in the below link.
use concatenate .
Regards,