I am loading in multiple CSVs and there has been a new column added to the new CSVs. I've noticed it will not load in the old ones that are missing the new column. Is there a way to ignore the error and still load in all the CSVs and just put nulls under the columns for the older CSVs?
A specified field must always exists - otherwise the load will fail with an error (errors could in general be skipped but in this case it won't be useful because the data won't be loaded).
The easiest way to handle it will be to move the new csv's into another folder and just using two different load-statements whereby the second load-statement get a concatenate(FirstTable) prefix.
Also quite easy is it to use a dummy-load and concatenate your csv there with something like:
FirstTable: load 'dummy' as Dummy autogenerate 0;
concatenate(FirstTable) load * from csv;
drop field Dummy; // maybe further fields if you don't need all from the csv
Also possible but with more efforts is to load the files with a filelist-loop and checking there within an in front load if this field exists to brach in different load-statements and/or reading all existing fields in variables and creating with them a load-statement on the fly.