Reload Logs Reader v 1.0

[Michael_Tarallo]

In an enterprise environment IT does not give access to the folders where the logs are stored, so developers have to ask to get the logs from IT which can be terribly slow and frustrating.

[Michael_Tarallo]

Hey guys - today I would like to introduce you to our latest guest blogger, Andrea Danielato.  Andrea has been with Qlik for 10 years and is a Senior Consultant. He has worked on customer projects acting as a developer, a project manager and an advisor for Qlik solutions. He has helped customers optimize app performance, define and implement best practices and infrastructure governance. He is heavily involved with teaching our valued partners how to deliver the best Qlik applications. On a personal note, Andrea likes to watch movies and loves to practice DIY. Please welcome me in joining Andrea to the guest blogger program - where today he will provide you with a solution to read RELOAD LOGS (attached).

Take it away Andrea!

Andrea:

Hello Qlik Community! You will be pleased to know that I have developed a solution for developers, that helps with access to Qlik Sense app reload logs for debugging purposes.

In an enterprise environment, it may be common where IT will not provide access to the file structure / folders where the logs are stored, so developers have to ask to get the logs from IT which can be terribly slow and frustrating.

To solve this I have developed a simple ODAG solution that can help on this topic.

I attach a document with a short manual of the solution including installation and a short vide that demonstrates the functionality of the solution.

Let me know what you think!

Reload Logs Reader V.1.0.zip

[Or]

Interesting! I've got a similar app built which reads the logs (including the log text) from the reload logs folder, as well as creating stripped-down versions without the timestamps so the SQL / load scripts can be copied by those don't have access to the script itself. It also means we can search for specific fields/code snippets from within QS itself.

Looks like there's stuff you're doing I could stand to learn from to improve on my version (or replace it outright). Thanks for sharing!

One added note - with the May 2021 QS release, it's possible for people to download the last log for each task directly from hub, assuming they're assigned the correct security rules and have access to the app itself, which also helps with giving people access to logs without having to give them direct access to the folders. Previously this could only be done by giving them QMC access to the Tasks screen, which was also an option but more cumbersome.

[Andrea-Danielato]

Thanks Or for your post.

The possibility to download the log from hub can be, from my experience, breaking some internal rules at customer side especially in Enterprise environment.

Anyway using this approach you can get only the last log and not the oldest ones where you can have additional details like a stopped execution and you can compare the timing to complete the reload.

[Or]

@Andrea-Danielato  Agreed - different use cases require different types of information, of course, and like I said your app has a lot I can learn from - I never thought to use ODAG for log reading.

Now if only we could get Qlik to have a setting that saves the last X logs, but also the last X logs which ended in an error, rather than only saving the last X (default is 4, I think) and then we lose track of the errors with apps scheduled to run frequently (or we save way too many logs). But that's for Ideation, I suppose.

[Andrea-Danielato]

@Or good point. We can post an idea on the ideation site about the last X logs but in the meantime I will make some thinking about checking the logs with error and save them in QVD for future investigation, with a time flag to delete the most old ones.

Thanks, I will add this to the next version!

Andrea

[mountaindude]

ODAG can indeed be used for all kinds of things - digging into logs probably wasn't one of the expected use cases - very useful nonetheless!!

If you want to keep track of your reloads and what caused them to fail, the Butler's alert emails can also be useful. 
Indeed, I am the creator of that tool, but as it's related but not competing I figured it's ok to mention it here.

The idea is that reload failures are tracked by means of a hook into Qlik Sense's logging framework, then forwarded to Butler. There the event is enriched with metadata such as reload logs, then sent as an email, Slack or Teams message, or as a webhook. 

The really nice part is that the alert email/Slack/Teams message can include (if so desired) the last few (again, configurable) rows in the reload log.
From there it's usually pretty easy to determine what caused the reload failure.

It can look like this:

Main Butler site is available at https://butler.ptarmiganlabs.com. Open source, of course.

Cheers,
Göran

[Andrea-Danielato]

@mountaindude very interesting approach to a common problem for developers and IT application management groups.

The only point that I can see in large enterprise is breaking security rules defined by the IT department. Generally asking to install something on a governed server is always a big fighting and a lot of discussion.

Anyway a very good idea of using different alerts systems to share the interesting log part.

Andrea

[Levente_Szittya]

Hi,

Could you please advise what shall be done with this error?

The dataconnection in the imported app does not work. I have tried to create a "PostgreSQL" data connection, but I do not know exactly what and where should I look for?

Thanks,

Levente

[Andrea-Danielato]

Hi Levente,

the Postgress_Idra library is a data connection created using the PostgreSQL data driver that point to the Qlik Sense database definited during the installation of Qlik Sense. You can ask to the IT that has managed the installation to get the parameters for the Postgere database, user and password.

Andrea