Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi guys,
I have sensor data coming from impala data base,and i have locations data coming from postgres data base. I have written very complex formulas using aggr() , if condition, and such other which causing a lot of performance issues on dashboard.
As of now we are dealing with 1-2 crores of records,but due to heavy computational formulas on each and every chart,it is taking some 50-60 secs to open the dashboard.
To make my dashboard fast, i want to handle those formulas in db side or scripting side, as i am getting data from two different sources when i try to write the formula,it is showing error like xxx field not found .
Could you plz help me .
You can do this through Resident Load.
For example:
Sensor_tmp:
Load
SensorId,
'Impala' as SensorSource,
SensorReading,
...
;
SQL Select
...
From Impala;
Concatenate(Sensor_tmp)
Load
SensorID as SensorId,
'Postgres' as SensorSource,
Reading as SensorReading,
...
;
SQL Select
....
From Postgres;
Sensor:
NoConcatenate Load
SensorId,
SensorSource,
SensorReading,
IF(SensorSource = 'Impala', SensorReading * 1.25, SensorReading) as Condition,
...
Resident
Sensor_tmp;
Drop Table Sensor_tmp;