topic Re: Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP? in Connectivity & Data Prep

Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

GrahamHannington — Tue, 28 Dec 2021 20:19:46 GMT

I help to develop software that forwards log data in JSON Lines format via TCP/IP. By "log data", I mean: time-based data; event data; data that consists of a time stamp and other fields.

I have configured other analytics platforms, Splunk and Elastic Stack, to ingest that data via TCP/IP. I have developed dashboards on those platforms to present and analyze that data.

Today, I received news that an organization wants to use that data in Qlik Sense Enterprise.

I have no experience using Qlik Sense.

Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

If so, could someone please point me to documentation on how to configure Qlik Sense Enterprise to do that?

If not, could someone please point me to documentation on how to extend Qlik Sense Enterprise to do that? Or an existing third-party extension?

The software that I help to develop does not currently offer a REST API. That is a possibility. First, however, I would prefer to investigate using the current forwarding method. Some reasons:

The software is currently designed to push data. It runs periodically (determined by the user: perhaps, hourly or daily). Depending on the period, each run can generate many megabytes of data.
Streaming JSON Lines over TCP/IP avoids having to "chunk" the data to fit a limited maximum size of an HTTP response.

Advice welcome.

Re: Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

GrahamHannington — Fri, 12 Jun 2020 14:55:32 GMT

I don't know:

Whether the term "ingest" is appropriate for Qlik Sense
Whether, or to what extent, Qlik Sense is a data store
Whether Qlik Sense is "fit for purpose" for analyzing time-based data (logs)

I do know something about:

The software that I help to develop
The data that it forwards
The forwarding method
How to use that data, and that forwarding method, with other analytics platforms
That an organization wants to use that data in Qlik Sense

I've only just started reading the Qlik Sense docs. I'll keep reading!

Re: Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

GrahamHannington — Fri, 12 Jun 2020 15:05:19 GMT

One possible approach: forward the JSON Lines via TCP/IP to a data store (database) that both:

Supports that ingestion method
and
Provides a connection method supported by Qlik Sense

MongoDB, maybe.

Re: Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

GrahamHannington — Sat, 13 Jun 2020 15:47:01 GMT

Or... as I'm doing now, forward the JSON Lines via TCP/IP to Elasticsearch, and then get the data from Elasticsearch via either:

I'm just reading about this... while I've used the Elastic REST APIs before, I've not yet used Qlik.

Re: Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

GrahamHannington — Mon, 15 Jun 2020 02:03:31 GMT

Or, if a proper REST API turns out to be overkill for this use case, simply write the JSON Lines (or CSV) to files on the remote platform (where the logs are generated), and serve those files over HTTP with a well-defined URL: e.g. for a daily summary, .../yyyy-mm-dd. For extra points, make the root directory (in this context; for this output) available over HTTP as a directory listing (for "discovery": so clients know what is available).