Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
"The answer is NO!"
That's what you nearly yell at someone who has been persistently asking you to the see the latest/greatest data values which are flowing through your Qlik Cloud Data Integration pipeline prior to the 15 minute cycles in which the project batches them up and moves to the Storage layer from the Landing layer.
[Be sure to turn on that Data Freshness Layer in your projects]
Apparently they have clout and they push for a meeting. During which time you explain that the Landing layer is intended to rapidly record changes, not report them. You even show them that the Patients table shows exactly what was "landed" during the full load. (Notice it does not show the changes you will see in the __ct table.)
And the Patients__ct table shows the changes that have occurred. Clearly neither is suitable for "Mr. Whiny Pants" to query from.
Just as you begin to think the case is closed some Dork sitting in the back row chimes up "Well what if you give them a view that will let them see the latest/greatest values." You explain that while you "could" do that, it would totally defeat the entire reason your team selected Qlik Cloud Data Integration in the first place. That with Qlik Cloud Data Integration everything is automated, you simply design and it automates the creation and application of code. That if you return to hand building views you would be building up a mountain of technical debt again.
Live Views
Admittedly, that idea of a Live view of the data for those occasions where the end user(s) really do need the latest/greatest sounds rather magical. Well start clapping now my friend because Qlik Cloud Data Integration already provides those "live" views for you. I'm not kidding.
The really cool thing about them is that they handle any transformation or rules you might have applied. Say if you defined that all tables in your Storage layer must have their column names prefixed with "GH_."
How clever is that? Notice that indeed it shows you the most recent value for Patient 2 that has been changed, and it also let's you know in the "hdr__store" column if the record is the CURRENT one that exists in the Storage layer, or if it is showing you information from the Change Table (CT.) And will tell you if the value is the same as what was Loaded (L) or it if reflects an Update (U) that occurred at some point. The regular Patients view still only shows what batches are applied during the 15 minute cycles you have defined.
After your predefined batch cycle triggers ... your Patients view will show the latest/greatest values
And notice that the Live view would now reflect that both records are CURRENT for the most recent applied batch of changes, while still showing that Patient 1's values are still what was originally loaded, but Patient 2's values reflect updates that occurred at some point.
Flexibility
In my previous post about Type 2 History I brought up the fact that the functionality was completely controllable and was super flexible. You could turn it on/off as needed by table. Guess what? The same applies for Live views. They default to be on for the Storage layer, but you are free to turn them off.
At the table/Dataset level that setting is inherited, but you are free to override the setting so that Qlik Cloud Data Integration either generates or does not generate the Live view.
Heads up
Be aware, as noted in the Qlik Help for Qlik Cloud Data Integration that Live views require more resources than standard views. So consider when/where they are used, and take their use in context with the batch schedule time to just put the CDC data into the Storage layer.
The Morale of the Story
When and if any persistent end user is begging you to see the latest/greatest values even before your scheduled batches are applied your answer can now be an emphatic Yes. The morale of that story is that story is obviously that you should read more posts from me, prior to walking in to meetings like that. 🤣
For you viewing pleasure
