Qlik Community

Qlik Associative Big Data Index Discussions

Ask a question or participate in a discussion about Qlik Associative Big Data Index here.

Highlighted
Partner
Partner

How to automatically index new data ?

Hello everyone,

In the commercial speach for QABDI, one of the main aspect of the product is the ability to have new data added to the datalake, automatically indexed by QABDI, and then available in a Qlik Sense app in live mode.

I have a working QABDI cluster, but I don't know how to configure this part.
If I add parquet files to one of my parquet folder, nothing happen, the new data is not indexed.

I've read this part of the documentation https://help.qlik.com/en-US/associative-big-data-index/June2019/Content/QABDI/qabdi-index-appending%... but:

- there is no start_gateway.sh script in the bastion pod, nor in the restapi pod
- the documentation mentions to run a curl command to https://localhost:8080/v1/bdi/triggersingleupdate, is this the only way to scan for new data ?

How can you automatically detect new data and index them ?

Thank you very much.

Best regards,

Thiebaud

Labels (2)
4 Replies
Partner
Partner

Re: How to automatically index new data ?

I believe the start_gateway.sh is the equivalent to start_rest_service.sh, which can be found in a pod at: 

 

/home/ubuntu/dist/runtime/scripts/rest_service/

 

 

Regarding your second point, you can either do it by calling the API or running the indexer manually. The idea is that you add the API call to your data pipeline workflow. So after you drop the file you call the API which then updates the index. 

If this doesn't fit within your data ecosystem, what you could do is write a small program that acts as a listener to your data folders and then calls the API if any new files are added/changed. 

Blog: WhereClause   Twitter: @treysmithdev
Partner
Partner

Re: How to automatically index new data ?

Thank you for your answer, I will try that.

 

EDIT: what I don't really understand is that in the doc https://help.qlik.com/en-US/associative-big-data-index/June2019/Content/QABDI/BDI-REST-API-data-appe...

it says:

 

To trigger a new index update without waiting for the index delay interval to elapse:
POST https://localhost:8080/v1/bdi/triggersingleupdate

 

 

So there is a "index delay interval" which should automatically trigger the update I guess... ?

 

Anyway, in the bastion pod, there is a start_rest_service.sh in /home/ubuntu/dist/runtime/scripts/rest_service/ but if I run it, I have the following error:

 

************************** Start ABDI Rest Service **************************
Create key and certificate
Generating RSA private key, 2048 bit long modulus (2 primes)
.................................+++++
..............................................................................................................................................................................................+++++
e is 65537 (0x010001)
Can't load /root/.rnd into RNG
140540902756800:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/root/.rnd
Done

 

The second time I run it, the error doesn't appear anymore, but I don't think it's starting any service. If I execute the ./service_manager.sh script, I don't see any "rest" service.

 

In the restapi pod, there is a abdi_rest_service script in /home/ubuntu/dist/rest_service, but it only displays the license when you execute it...

 

Employee
Employee

Re: How to automatically index new data ?

Hi

Currently new data can be added incrementally via the REST call which will append the new parquet files into the indexing process, an error on the help site being changed it should read:

/home/ubuntu/dist/runtime/scripts/rest_service/start_rest_service.sh

for the service name

thanks

Ian
Partner
Partner

Re: How to automatically index new data ?

Hi Ian,

Thanks for your answer.

I've started the rest service using the sh script.

Now when I exec into the bdi-restapi pod and execute the curl command, I have the error below:

root@bdi-restapi-65d48d95d4-m75gc:/home/ubuntu# curl -k -X POST -H "Content-Type: application/json" -d "{\"schema_name\":\"bdidata\"}"  "https://localhost:8080/v1/bdi/triggersingleupdate"
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

 

 Do you know how to fix it ?

Thanks