topic Re: Reading/Writing from Log Stream end point in Qlik Replicate

Reading/Writing from Log Stream end point

deeptimathew — Tue, 14 Feb 2023 17:27:07 GMT

Morning, am trying to set up a below test case. Did anyone come across a use case I described below?

QLIK Replicate Server 1

1. Read from Oracle databases

2. Write to Log Stream Target End Point

QLIK Replicate Server 2 (Remote on a different network)

1. Read from QLIK Replicate Server 1 Log Streams

2. Write changes from Log Streams to Target Oracle end point

Thank You

Deepti Mathew

Re: Reading/Writing from Log Stream end point

Dana_Baldwin — Tue, 14 Feb 2023 17:48:14 GMT

Hi @deeptimathew

When using Log Stream, both the staging/parent task and the replication/child task must reside on the same Qlik Replicate server. This is because you need to define the staging task in the source endpoint of the replication task.

https://help.qlik.com/en-US/replicate/November2022/Content/Replicate/Main/Log%20Stream%20Staging/ls4.htm

If a second Qlik Replicate server needs to be used, the feature you would use is File Channel. More information can be found here:

https://help.qlik.com/en-US/replicate/November2022/Content/Replicate/Main/File%20Channel/file_channel.htm#Using%20the%20Qlik%20Replicate%20file%20channel

Thanks,

Dana

Re: Reading/Writing from Log Stream end point

deeptimathew — Tue, 14 Feb 2023 19:02:40 GMT

Hey Dana - Thank you for your response.

Thank You

Deepti

Re: Reading/Writing from Log Stream end point

Dana_Baldwin — Tue, 14 Feb 2023 19:12:55 GMT

Hi @deeptimathew

If you don't have any further questions, please mark my response as a solution.

Thanks!

Dana

Re: Reading/Writing from Log Stream end point

deeptimathew — Tue, 14 Feb 2023 19:18:56 GMT

Hey Dana - Just to confirm the second Qlik Replicate server can be on a completely different network correct? The reason am asking is the client is external and wont be on the same network as the first Qlik Replicate server.

Thank You

Deepti

Re: Reading/Writing from Log Stream end point

Dana_Baldwin — Tue, 14 Feb 2023 19:26:33 GMT

Hi @deeptimathew

If you want to push the data to an endpoint that is not in your LAN, use the File Transfer Service to send the files created in the local task to the remote location.

https://help.qlik.com/en-US/replicate/November2022/Content/Global_Common/Content/SharedEMReplicate/Server%20Settings/file_transfer_service.htm

Remote task #Remote task | Qlik Replicate Help

Thanks,

Dana

Re: Reading/Writing from Log Stream end point

deeptimathew — Wed, 15 Feb 2023 20:56:15 GMT

Hey Dana - Once FTS is set up on the Remote Qlik server, port 9090 should be open from the Local Qlik server to be able to communicate to the File Transfer Service set up on the Remote Qlik server as well. Is my understanding correct?

Thank You

Deepti

Re: Reading/Writing from Log Stream end point

SwathiPulagam — Wed, 15 Feb 2023 21:27:38 GMT

Hi @deeptimathew,

Yes if you use the default port for FTS then you are correct, the port 9090 should be open from the Local Qlik server.

Thanks,

Swathi

Thanks,

Swathi

Re: Reading/Writing from Log Stream end point

Dana_Baldwin — Wed, 15 Feb 2023 22:19:52 GMT

Hi @deeptimathew

Yes, by default port 9090 is used and has to be open on both Replicate servers.

Thanks,

Dana

Re: Reading/Writing from Log Stream end point

deeptimathew — Thu, 23 Feb 2023 15:24:53 GMT

Hey Dana - Morning, I was able to enable the FTS service and set up a task on Qlik Replicate 1 to write files to Qlik Replicate 2 using File Channel target end point. I see files generating on Qlik Replicate 2 as well. I setup a task on Qlik Replicate 2 to read the files and apply changes in an Oracle target end point. Here is where I see issues, I don't see the task picking up the generated files on Qlik Replicate 2. The task doesn't fail or give errors, it keeps looking for FC files on Qlik Replicate 2. Anything I might be missing? Any pointers to what I can look for?

Thank You

Deepti

Re: Reading/Writing from Log Stream end point

Dana_Baldwin — Thu, 23 Feb 2023 18:01:39 GMT

Hi @deeptimathew

This may require a closer look. Could you please open a support case with the diagnostics packages for both tasks?

Thanks,

Dana

Re: Reading/Writing from Log Stream end point

Steve_Nguyen — Mon, 27 Feb 2023 13:55:40 GMT

@deeptimathew

if i read this correctly.

server 1:

Oracle -> Filechannel target on Server 2

server 2

Filechannel source (filechannel target from server1) to oracle

but you setup as logstream task,

====

notes logstream can not use file channel limitation :

File Channel cannot be used as a source for a Log Stream Staging task.

https://help.qlik.com/en-US/replicate/November2022/Content/Replicate/Main/Log%20Stream%20Staging/ls3.htm

Re: Reading/Writing from Log Stream end point

deeptimathew — Tue, 28 Feb 2023 22:13:51 GMT

Hey Dana - I brought this up with qlik professional services contact am working with.

This is not possible with the application. File Channel is the way to go when reading CDC changes from Replicate Server 1 and writing to another Replicate Server 2 before processing the changes in a target Oracle end point from Replicate Server 2.

Thank You

Deepti

Re: Reading/Writing from Log Stream end point

deeptimathew — Tue, 28 Feb 2023 22:17:22 GMT

Hey Steve - Correct, your answer is accurate.

Thank You

Deepti

Re: Reading/Writing from Log Stream end point

deeptimathew — Tue, 28 Feb 2023 22:23:52 GMT

Follow up question on File Channel, is it possible to read from File Channel source end point and write to a File Channel target end point? Is it a valid use case with Qlik Replicate?

Thank You

Deepti

Re: Reading/Writing from Log Stream end point

Dana_Baldwin — Wed, 01 Mar 2023 00:01:22 GMT

Hi @deeptimathew

I am not sure I understand the use case. The local File channel task reads from a database source and writes to the file channel. The remote file channel task reads the file channel files as a source and writes to a database. I suppose it could write to another file channel, but I am not sure that is feasible since you would only be involving a third Replicate server for no obvious reason... Please elaborate.

Thanks,

Dana

Re: Reading/Writing from Log Stream end point

Steve_Nguyen — Wed, 01 Mar 2023 18:16:23 GMT

@deeptimathew :

Follow up question on File Channel, is it possible to read from File Channel source end point and write to a File Channel target end point? Is it a valid use case with Qlik Replicate?

ans : no

that is where File Channel read once from DB source and write to many FC .

Re: Reading/Writing from Log Stream end point

deeptimathew — Tue, 18 Apr 2023 19:29:25 GMT

Hey Dana - The use case I was trying to solve is sending File Channel files to an external company. As we cant directly communicate to the external company directly I was trying to find a way to get them the CDC files generated by File Channel target end point. Our team worked with Qlik and we narrowed down 2 approaches

1. Oracle Source - File Channel target using FTS

2. Oracle Source - File Channel target , use SFTP to copy files generated by File Channel to a different location, external company pulls from the location files are dropped to consume using Qlik Replicate on their end

The second approach is due to security limitations between both companies

Thank you for your suggestions, it helped me understand how the tool works.

Deepti