How to do calculations on scripting side .when data coming from two different sources

[srg99999]

 Hi guys,

I have sensor data coming from impala data base,and i have locations data coming from postgres data base. I have written very complex formulas using aggr() , if condition, and such other which causing a lot of performance  issues on dashboard.

As of now we are dealing with 1-2 crores of records,but due to heavy computational formulas on each and every chart,it is taking some 50-60 secs to open the dashboard. 

To make my dashboard fast, i want to handle those formulas in db side or scripting side, as i am getting data from two different sources when i try to write the formula,it is showing error like xxx field not found .

Could you plz help me .

[treysmithdev]

You can do this through Resident Load.

For example:

Sensor_tmp:
Load
    SensorId,
    'Impala' as SensorSource,
    SensorReading,
    ...
;
SQL Select 
...
From Impala;

Concatenate(Sensor_tmp)
Load
    SensorID as SensorId,
    'Postgres' as SensorSource,
    Reading as SensorReading,
    ...
;
SQL Select
....
From Postgres;

Sensor:
NoConcatenate Load
    SensorId,
    SensorSource,
    SensorReading,
    IF(SensorSource = 'Impala', SensorReading * 1.25, SensorReading) as Condition,
    ...
Resident
    Sensor_tmp;

Drop Table Sensor_tmp;