Same file having different delimiter on a daily basis

[vardhancse]

Hi,

I have one issue is that, on a daily basis I will get one new CSV file. Need to incremental load and append to existing data/QVD.

Its pretty straight forward, However issue is that:

File Name1: abc_2019-01-08.csv

File Name2: abc_2019-01-19.csv

Each file is being generated/extracted with different delimiter, say like

abc_2019-01-08.csv is having ';' as delimiter

abc_2019-01-09.csv is having ','  as delimiter

Can any one let me know is there a way to load based on available delimiter but not static delimiter.

We have some 100 files, need to be loaded on a daily basis, each file is having its own delimiter, something we were unable to change from source system

[zzyjordan]

the delimiter is randomly assigned to a csv file or for the same process, it always uses the same delimiter? if that's the case, probably in the loading script, load the files with same delimiter in a loop and then concatenate them together.
ZZ

[vardhancse]

Issue is delimiter is randomly assigned, not always one for same file.

For each day generating with different delimiter

[marcus_sommer]

One possibility is to use ERRORMODE to ignore errors and just to continue with the next step and/or to query the errors and to react on them. Another way would be to pre-load the first record as one fieldvalue and to detect within a variable for example with substringcount() which delimiter is used and then using this variable within the fileformat-statement.

- Marcus

[vardhancse]

Hi Marcus,

Thanks for your response.

I can not ignore errors, I can go with your second option but issue is that, not only delimiter is different other factors like 'MSG' is different/dynamic across multiple CSV extracts.

some times its like 'no quotes' and 'msg'.

Can please see to below:

(txt, utf8. embedded labels, delimiter is ';', no quotes);

(txt, codepage is 28591. embedded labels, delimiter is ',',  msq);

Would you be able to help me to sort my issue

[marcus_sommer]

Then you need to check your other dynamically changing attributes too - most of it should be checkable with my mentioned approach of loading the first record and applying various string-functions to detect the used attributes. I think the following will give you a good starting point to implement such logic: Load-CSV-files-with-different-format

- Marcus

[vardhancse]

Hi Marcus,

Thank you, will try now.