Connectivity & Data Prep

[GrahamHannington]

I help to develop software that forwards log data in JSON Lines format via TCP/IP. By "log data", I mean: time-based data; event data; data that consists of a time stamp and other fields.

I have configured other analytics platforms, Splunk and Elastic Stack, to ingest that data via TCP/IP. I have developed dashboards on those platforms to present and analyze that data.

Today, I received news that an organization wants to use that data in Qlik Sense Enterprise.

I have no experience using Qlik Sense.

Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

If so, could someone please point me to documentation on how to configure Qlik Sense Enterprise to do that?

If not, could someone please point me to documentation on how to extend Qlik Sense Enterprise to do that? Or an existing third-party extension?

The software that I help to develop does not currently offer a REST API. That is a possibility. First, however, I would prefer to investigate using the current forwarding method. Some reasons:

• The software is currently designed to push data. It runs periodically (determined by the user: perhaps, hourly or daily). Depending on the period, each run can generate many megabytes of data.
• Streaming JSON Lines over TCP/IP avoids having to "chunk" the data to fit a limited maximum size of an HTTP response.

Advice welcome.

[GrahamHannington]

I don't know:

• Whether the term "ingest" is appropriate for Qlik Sense
• Whether, or to what extent, Qlik Sense is a data store
• Whether Qlik Sense is "fit for purpose" for analyzing time-based data (logs)

I do know something about:

• The software that I help to develop
• The data that it forwards
• The forwarding method
• How to use that data, and that forwarding method, with other analytics platforms
• That an organization wants to use that data in Qlik Sense

I've only just started reading the Qlik Sense docs. I'll keep reading!

[GrahamHannington]

One possible approach: forward the JSON Lines via TCP/IP to a data store (database) that both:

• Supports that ingestion method
  and
• Provides a connection method supported by Qlik Sense

MongoDB, maybe.

[GrahamHannington]

Or... as I'm doing now, forward the JSON Lines via TCP/IP to Elasticsearch, and then get the data from Elasticsearch via either:

• Elasticsearch REST API
  or
• Elasticsearch ODBC driver

I'm just reading about this... while I've used the Elastic REST APIs before, I've  not yet used Qlik.

[GrahamHannington]

Or, if a proper REST API turns out to be overkill for this use case, simply write the JSON Lines (or CSV) to files on the remote platform (where the logs are generated), and serve those files over HTTP with a well-defined URL: e.g. for a daily summary, .../yyyy-mm-dd. For extra points, make the root directory (in this context; for this output) available over HTTP as a directory listing (for "discovery": so clients know what is available).