Task Scheduling For Full Load +CDC Task in Qlik Cloud Data Integration

[Chirag_]

Hi Team,

While testing the task scheduling scenario, I found that I'm unable to schedule a Full Load + CDC task. Could any experts assist in confirming if this is possible?

Regards,

Chirag

[TimGarrod]

Hi, for full load + CDC tasks those tasks are always-on when running in your on-premise data gateway.    Once you run the task the full load runs to instantiate the target and then cdc is running in real-time capture mode.    CDC tasks need to run all the time to capture your source changes and deliver the data to the target. 

If you were to schedule the task for every X hours, but you removed source log files by backups etc. then the cdc process would not be able to continue. 

Could you please explain the scenario in which you want to schedule a CDC task ? 

[Chirag_]

Hi Tim,

we are having a scenario where  : 

• our source system faces a high consumption lets say  from 10 am to 6 pm , where we would like to stop the task , and during off peak hours we would like to resume the cdc task. so here we wanted to schedule the cdc task to start/ stop as per our requirement 

what my understanding was that while using a gateway it will always capture cdc and accumulate changes in the gateway and append once the task is resumed 

i would like to know what's your thoughts on this scenario could be solved 

Regards,

Chirag 

[TimGarrod]

Hi - thats a misunderstanding of how the task works. 

When you use a Replication or Landing task, the task needs to be running in order to capture changes.   Those changes are sent to the target, but you can adjust the timeframe of how quickly they are delivered.   That timeframe however is typically not meant to be set at more than 15 minutes as the changes are cached in the gateway server. 

So a landing or replication task must be running in order to capture changes. 

If you have a high volume source, you want the gateway to be capturing changes throughout the day in order to 'keep up' - otherwise you are asking it to try to read through 8 hrs of high volume logs at the end of the day - giving it a massive bottleneck to run through. 

Is the concern the delivery or the capture? 

It sounds like you are saying you want capture throughout the day, but delivery to the target to be scheduled ? 
What is your target? 

Have you looked at Pipeline projects with LANDING and STORAGE, where the storage can be scheduled (landing will deliver data to Snowflake for example throughout the day as a 'staging' for the persisted STORAGE task to process. 

[Chirag_]

Hi Tim,

 It sounds like you are saying you want capture throughout the day, but delivery to the target to be scheduled ?  - Yes, that’s exactly what I’m trying to achieve, but through a replication task.

In my previous experience with Qlik Replicate, I was able to schedule the CDC (Change Data Capture) task, as shown in the attached reference.

I wanted to Achieve the Same in QCDI , but what I observed in QCDI that we could only Schdule a Full Load Task. Do you think of any way we could achieve the same using Replication task in QCDI?

Regards,

Chirag

[TimGarrod]

You can use a Qlik Application Automation to STOP and then START a task and schedule it. 

[Chirag_]

Hi Tim,

Thanks for suggestion ,
we want to try the automation application

It would be helpful if you can provide some resources from where we can understand how automation application can be used for scheduling and will let you know the results.

Best Regards,

Chirag