topic Re: File as a source config in Qlik Replicate

File as a source config

allann — Thu, 20 Oct 2022 15:17:15 GMT

We're trying to setup a task with a csv file as a source. I've defined all the tables which are located on the same server at /qlik/source_files/cisco_cucm/test Then I defined the target folder path as /qlik/target_files/cisco_cucm/ but I'm getting an error saying folder does not exist. There are currently no files there if that matters. Any advice?

Re: File as a source config

Alan_Wang — Thu, 20 Oct 2022 15:56:57 GMT

Hi @allann

Can you check if the user running the Replicate services has permission to create directories and files? The user should be able to create them for you. You can also create them manually and use the existing file paths.

Re: File as a source config

Heinvandenheuvel — Thu, 20 Oct 2022 16:15:16 GMT

You mention file as source? flullload + cdc? -

It looks like the target is also a file? (for testing perhpas)

Why are you using single slash ( / ) device-directory name in the text, but double slash ( // ) in the pictures?

Why are you using forward slash in the the text, but backward slash in the pictures?

Why are the change processing files pointed to ina target named forder?

Personally I _hate_ pictures as 'evidence' and prefer 1000 words that I can cut and past and have contexts.

Please provide that task JSON (export) and the task LOG file such that all information is there, not just the bits and pieces you believe to be relevant.

Cheers,

Hein

Re: File as a source config

allann — Thu, 20 Oct 2022 17:52:29 GMT

This is my first time trying to set a task up with file as a source. In searches, I saw double slash so that's what I tried after forward slash didn't work. I just changed to match my original text (/). Now seeing "When the "Apply Changes" and/or "Store Changes" replication options are enabled,the ordinal position of the "Data start" header column cannot be zero."

This is a full load file, there will be multiple happening once a month.

Re: File as a source config

Heinvandenheuvel — Thu, 20 Oct 2022 18:32:32 GMT

>> "Data start" header column cannot be zero."

So make it '1' ?

maybe more later - if you haven't figured it out by then.

Re: File as a source config

allann — Thu, 20 Oct 2022 19:41:06 GMT

That didn't work. Is it an issue that we have table names in the header and data type in the 2nd row?

Re: File as a source config

Heinvandenheuvel — Fri, 21 Oct 2022 00:43:34 GMT

Single forward slashes is what you need as the LOG file provided (thank you) shows you are running on Linux.

Like I said - words, bytes. No bits in pictures. The inclusion of data.png is a good attempt. However this appears to be Excel's 'interpretation' of the data. We cannot see whether there are actually commas as claimed or just a newline as terminator like a Linux file might have and not the carriagereturn \r) and a newline(\n), also referred to as CR/LF which all to often the case for a windows file. Another thing - I'm ready to bet you a nice four pack of beers (six pack 12oz are now often the lower quality beers) - 4x16oz is the nice stuff 🙂 that your data does NOT contain 6.89E+08 as the picture suggests - so what can we trust? You see my issue with the picture? Use VIM or XD or something to verify the format please.

Next that INT1 datatype - that's a 1 byte integer and thus a max value range -128 to 127. That's not what you wanted is it now? RTFM !!

The files as shown needs on the source endpoint the setting "Ignore records: ... Header:... 2 in order to skip te column names an datatype lines.

I suspect you really only want a full-load. Change the task and try that first!

There are no (recognizable) CDC drive header column such as Operations, Table names, Timestamps, User names. So all you could CDC would be INSERTS to a single table. RTFM!

Do NOT have the full-folder be a sub-directory of the CDC folder. That's confusing and will not work. Make them be 'parallel'.

Be sure to carefully read up on Reference files. Then read the whole chapter "8.6 Using a file as a source". Then re-read the section on Reference files. Then re-read the whole chapter, but slowly.

Now try again! I suspect it will work now.

If not increase the logging level for IO and SOURCE_CAPTURE and see if that clarifies better where the instructions given to Replicate did not match the reality.

Hein.

Re: File as a source config

allann — Wed, 26 Oct 2022 18:18:49 GMT

Looks like the issue was the task type and extra columns not matching the source file.

Re: File as a source config

Heinvandenheuvel — Wed, 26 Oct 2022 19:29:03 GMT

>> Looks like the issue was the task type and extra columns not matching the source file.

Thanks for the feedback confirming my suggestion:

"I suspect you really only want a full-load. Change the task and try that first!"

Hein