Qlik Community

Ask a Question

Connectivity & Data Prep

Discussion board where members can learn more about Qlik Sense Data Connectivity.

Announcements
LINKEDIN LIVE: Democratizing data to enhance customer-centricity. JULY 29TH REGISTER TODAY
cancel
Showing results for 
Search instead for 
Did you mean: 
GrahamHannington
Contributor
Contributor

Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

I help to develop software that forwards log data in JSON Lines format via TCP/IP. By "log data", I mean: time-based data; event data; data that consists of a time stamp and other fields.

I have configured other analytics platforms, Splunk and Elastic Stack, to ingest that data via TCP/IP. I have developed dashboards on those platforms to present and analyze that data.

Today, I received news that an organization wants to use that data in Qlik Sense Enterprise.

I have no experience using Qlik Sense.

Can Qlik Sense Enterprise ingest JSON Lines via TCP/IP?

If so, could someone please point me to documentation on how to configure Qlik Sense Enterprise to do that?

If not, could someone please point me to documentation on how to extend Qlik Sense Enterprise to do that? Or an existing third-party extension?

The software that I help to develop does not currently offer a REST API. That is a possibility. First, however, I would prefer to investigate using the current forwarding method. Some reasons:

  • The software is currently designed to push data. It runs periodically (determined by the user: perhaps, hourly or daily). Depending on the period, each run can generate many megabytes of data.
  • Streaming JSON Lines over TCP/IP avoids having to "chunk" the data to fit a limited maximum size of an HTTP response.

Advice welcome.

Labels (3)
4 Replies
GrahamHannington
Contributor
Contributor
Author

I don't know:

  • Whether the term "ingest" is appropriate for Qlik Sense
  • Whether, or to what extent, Qlik Sense is a data store
  • Whether Qlik Sense is "fit for purpose" for analyzing time-based data (logs)

I do know something about:

  • The software that I help to develop
  • The data that it forwards
  • The forwarding method
  • How to use that data, and that forwarding method, with other analytics platforms
  • That an organization wants to use that data in Qlik Sense

I've only just started reading the Qlik Sense docs. I'll keep reading!

GrahamHannington
Contributor
Contributor
Author

One possible approach: forward the JSON Lines via TCP/IP to a data store (database) that both:

  • Supports that ingestion method
    and
  • Provides a connection method supported by Qlik Sense

MongoDB, maybe.

GrahamHannington
Contributor
Contributor
Author

Or... as I'm doing now, forward the JSON Lines via TCP/IP to Elasticsearch, and then get the data from Elasticsearch via either:

I'm just reading about this... while I've used the Elastic REST APIs before, I've  not yet used Qlik.

GrahamHannington
Contributor
Contributor
Author

Or, if a proper REST API turns out to be overkill for this use case, simply write the JSON Lines (or CSV) to files on the remote platform (where the logs are generated), and serve those files over HTTP with a well-defined URL: e.g. for a daily summary, .../yyyy-mm-dd. For extra points, make the root directory (in this context; for this output) available over HTTP as a directory listing (for "discovery": so clients know what is available).