Too much time || Data load || Qliksense Script ||

[mramitbhandari17]

Hi,

I have built a connection ODBC connection Elastic search with QlikSense to fetch the data. 

Multiple customers data of Firewall and we have each day 2-4 billion data for each customer in Elastic index.

SQL query in Qliksense to get the data but it is taking too much time 10-12 hours for data load in Qlik sense.

Is any other way i can work upon. Moreover I have used select statement with selected columns.

-----------------------------------------------------------------------------------------------------------------------------------

Below is the example of query.

Select date, customer.name, customer.id, vendor.name, check.name, location, max(cpu),min(cpu),avg(cpu), max(mem),min(mem), avg(mem)

from

table_name

where customer.name IN ('123', '12323', .. ...,..) and (check.name LIKE *cpu* OR check.name LIKE *mem*)

group by date, customer.name, customer.id, vendor.name, check.name, location

-----------------------------------

kindly suggest to get better output in minimum time of data load.

[Muthukumar_77]

Hi,

Don't reload the query with aggregation. Below like this,

Select date, 

customer.name, 

    customer.id, 

    vendor.name, 

    check.name, 

    location,

    cpu, 

    mem

from table_name

where customer.name IN ('123', '12323', .. ...,..) and (check.name LIKE *cpu* OR check.name LIKE *mem*);

Store as QVD format;

Do all transformation after store as QVD format.

And also implement incremental load based on date.

 Refer this: https://community.qlik.com/t5/Design/Overview-of-Qlik-Incremental-Loading/ba-p/1466780