Skip to main content
Announcements
Accelerate Your Success: Fuel your data and AI journey with the right services, delivered by our experts. Learn More
cancel
Showing results for 
Search instead for 
Did you mean: 
daturpin
Partner - Creator II
Partner - Creator II

How to load and transform data that is already loaded in Data Manager

This seems like it should be very easy, but I am struggling.

I have lots of tables loaded into my Qlik app from Databricks, and I would like to write a NEW table from some data without connecting to databricks again.

I tried everything on the help page https://help.qlik.com/en-US/qlikview/May2023/Subsystems/Client/Content/QV_QlikView/Scripting/load-da...

And I have used inline and resident loads previously, but only when connected to databricks.

Specifically, I have a table called 'actions' with an id, latitude, longitude, altitude, time, and other data. I want to be able to transform that existing data to iterate positions from the table. In the table, each id might have one, or a hundred position and time entries. Ultimately, I will need to iterate through the table to find the times closest to a variable time that I specify in a slider, so that I can iterate between them to find the specific positions at the given time....

But that is several steps away. Right now I am struggling to even load a new table with data from a table that is already loaded into data manager.

Thank you!

Labels (4)
1 Solution

Accepted Solutions
daturpin
Partner - Creator II
Partner - Creator II
Author

Okay, I figured out a way to do what I want on QlikSense. Added here in case there is anyone else who wants to create a macro in QLikSense for transforming tables. It is a multi-step process.

1) Create a data layer. In my case, I have a table which contains physical actions a vessel at sea can execute. To interpolate positions on a globe, all I needed was the actions with position and temporal data. So in a full reload, I load partial data from this table into a temporary table and then use the STORE command to place this data in a QVD file. During a partial reload, I then pull data from this QVD for calculations.

2) Execute macro using Vizlib. Previously I had used VizLib slider to set the "now" time for display, but this doesn't have macro execution. I found the better method was to use VizLib Input Form. A slider is available as an input. So I made a form with a slider, and coded the slider the same way as the previous one, where the min and max were based on the Max() and Min() times from the above table, and default being the variable vslider_time that will be set next. Set the VlizLib Writeback settings to write to a serverless QVD file.

3) Now that code can be executed, within the VizLib Writeback, set the actionable variable but modding the code:

temp1:
REPLACE LOAD
slider_time
From "lib://AppExports/update_slider_time.qvd" (qvd, utf8);
LET vslider_time = peek('slider_time', -1, 'temp1');

Then, at the end of this code, call a subroutine and pass the variable

CALL iterate_position(vslider_time);

4) Now the subroutine can chew on the data and save positions for everything. I am still working on this, but this code does an okay job:

sub iterate_position(vslider_time)
TRACE start of iterate_position subroutine;
	[iterated_position]:
    REPLACE LOAD 
	*
 	Inline [
'units_id','iterated_position.lat','iterated_position.lon'];
    [temp_actions]:
	LOAD
    actions_id,
    units_id,
    latitude,
    longitude,
    game_time
	FROM [$(vQVDPath)temp_actions.QVD] (qvd);
	LET NumRows=NoOfRows('temp_actions'); //Used to set the number of iterations to read the table
	SET IsEmpty=If(Isnull($1),'Yes','No'); //Used to determine if a Field is Null
	FOR i=0 to $(NumRows)-1
		LET vmaxtime = 45620; //Max(game_time);
		LET vmintime = 45520; //Min(game_time);
		LET vunits_id=Peek('units_id',$(i),'temp_actions');
		FOR j=i to $(NumRows)-1
			LET vunits_id_inner=Peek('units_id',$(j),'temp_actions');
        	IF vunits_id = vunits_id_inner then        	
    			LET vgame_time=Peek('game_time',$(j),'temp_actions');
				LET vlat=Peek('latitude',$(j),'temp_actions');
                LET vlon=Peek('longitude',$(j),'temp_actions');
            	//check to see if the game_time on this row is nearer to slider_time; iteratively should end up with time brackets
            	IF vgame_time >= $(vslider_time) and vgame_time <= vmaxtime then
            		LET vmaxtime = vgame_time;
                    LET vlatmax = vlat;
                    LET vlonmax = vlon;
            	END IF;
          		IF vgame_time <= $(vslider_time) and vgame_time >= vmintime then
            		LET vmintime = vgame_time;
                    LET vlatmin = vlat;
                    LET vlonmin = vlon;
            	END IF;
        		// Warning: This is a STRAIGHT interpolation. Replace this with a Vincenty algorithm to make it accurate over long distances!
            	LET iterated_lat = ((vslider_time - vmintime) / (vmaxtime - vmintime)) * (vlatmax - vlatmin) + vlatmin;
            	LET iterated_lon = ((vslider_time - vmintime) / (vmaxtime - vmintime)) * (vlonmax - vlonmin) + vlonmin;
            END IF;
        NEXT j; 
        [iterated_position]:
    	ADD ONLY LOAD *
        Inline [
'units_id','iterated_position.lat','iterated_position.lon'
$(vunits_id), $(iterated_lat), $(iterated_lon)];
		//we should now have iterated through the table only finding ONE unit with time bracketed between the two nearest points. (These points might be vmaxtime or vmintime)	
	NEXT i;
    If IsPartialReload() then 
    	STORE iterated_position into $(vQVDPath)iterated_position.qvd (qvd);
    End If
end sub

5) And finally, in the visualization I use the interpolated positions in my map, and they can move. It still isn't quote as fast as I would like but a good partial solution.

View solution in original post

9 Replies
Or
MVP
MVP

This should be fairly simple:

NewTableName:

Load Field1, Field2

Resident OriginalTableName;

 

daturpin
Partner - Creator II
Partner - Creator II
Author

Yes, that seems like that should work, but when I do that, and load data, I get the error "table 'actions' not found"

So I thought maybe it was because of the FILO ordering of the load command, so I tried placing it under the Auto Generated Section, and then after waiting for everything to reload from Databricks, I then get the error "Field 'id' not found"

And I don't understand either case. The table exists, I can display the data in my map, and the Field name is correct, and I don't want to reload data.

Or
MVP
MVP

I wouldn't know what's in the auto-generated section, as I never use those (and I'd advise against using them as much as possible - it's better to know what's in the script).

Note that names are case-sensitive, and you need to use whatever name Qlik is familiar with at the time of the load, e.g.

Table1:

Load Field1 as Hello, Field2 as World From SomeTable;

Table2:

Noconcatenate Load Hello, World Resident Table1;

If you use Qualify, your fields are likely to instead be named Table1.Hello (in the above example) as the table name is appended to the field name.

 

 

daturpin
Partner - Creator II
Partner - Creator II
Author

I am, unfortunately, not good enough at these transformations to ignore the functionality of the Data Manager GUI; at least for now.

But I think that did solve part 1. I found that the Field name of 'id' was actually called [id-id] in the Editor, possibly because it is a common link between various tables?

Can you help with the second part: Is it possible to conduct a transformation on a table that takes in a variable from outside the load editor?

There are many different 'id' used to indicate one moving item in 4D space. Can I iterate though a table, get the closest time values for a unique id that bracket a variable 'slider_time', and use that to generate a SINGLE row for each unique id of the iterated position?

Or
MVP
MVP

What you're asking for is unfortunately not possible for a traditional app - you're trying to combine data load functionality (table transformation) with front-end functionality (variable controlled by a slider). Maybe if you allow users to reload the app after setting the variable, but that's not typically a desirable option.

You might be able to write an appropriate formula on the front end of things to achieve a similar result, depending on what you mean by "closest time values". You would use ID as your dimension and some sort of formula to get the max / min / firstsortedvalue / rank matching your definition of "closest". I'd suggest elaborating on what exactly you want to achieve in this regard in a new Community post with an appropriate subject - that'll hopefully get the right eyeballs on your post and get you more specific help.

daturpin
Partner - Creator II
Partner - Creator II
Author

Yeah, that is what I was afraid of.

I actually already have the code to do what I want within Databricks using Python, but I wanted to see if I could do a front-end on Qlik to make it faster...  Because otherwise I am stuck sending points to Databricks, waiting for a job to run, getting the output, and reloading the app to see the change, which is not ideal.

Or
MVP
MVP

In this regard, the only potential alternative I see is writing something on the front end in Qlik. As I said, this may well be feasible depending on the specific requirement. It might be something along the lines of:

Dimension: ID

Expression: FirstSortedValue(Date, fabs(Date - vSomeVariable))

But as I said, without knowing the specifics it's hard to try and tell. May well be worth posting separately as I suggested.

daturpin
Partner - Creator II
Partner - Creator II
Author

Okay, I figured out a way to do what I want on QlikSense. Added here in case there is anyone else who wants to create a macro in QLikSense for transforming tables. It is a multi-step process.

1) Create a data layer. In my case, I have a table which contains physical actions a vessel at sea can execute. To interpolate positions on a globe, all I needed was the actions with position and temporal data. So in a full reload, I load partial data from this table into a temporary table and then use the STORE command to place this data in a QVD file. During a partial reload, I then pull data from this QVD for calculations.

2) Execute macro using Vizlib. Previously I had used VizLib slider to set the "now" time for display, but this doesn't have macro execution. I found the better method was to use VizLib Input Form. A slider is available as an input. So I made a form with a slider, and coded the slider the same way as the previous one, where the min and max were based on the Max() and Min() times from the above table, and default being the variable vslider_time that will be set next. Set the VlizLib Writeback settings to write to a serverless QVD file.

3) Now that code can be executed, within the VizLib Writeback, set the actionable variable but modding the code:

temp1:
REPLACE LOAD
slider_time
From "lib://AppExports/update_slider_time.qvd" (qvd, utf8);
LET vslider_time = peek('slider_time', -1, 'temp1');

Then, at the end of this code, call a subroutine and pass the variable

CALL iterate_position(vslider_time);

4) Now the subroutine can chew on the data and save positions for everything. I am still working on this, but this code does an okay job:

sub iterate_position(vslider_time)
TRACE start of iterate_position subroutine;
	[iterated_position]:
    REPLACE LOAD 
	*
 	Inline [
'units_id','iterated_position.lat','iterated_position.lon'];
    [temp_actions]:
	LOAD
    actions_id,
    units_id,
    latitude,
    longitude,
    game_time
	FROM [$(vQVDPath)temp_actions.QVD] (qvd);
	LET NumRows=NoOfRows('temp_actions'); //Used to set the number of iterations to read the table
	SET IsEmpty=If(Isnull($1),'Yes','No'); //Used to determine if a Field is Null
	FOR i=0 to $(NumRows)-1
		LET vmaxtime = 45620; //Max(game_time);
		LET vmintime = 45520; //Min(game_time);
		LET vunits_id=Peek('units_id',$(i),'temp_actions');
		FOR j=i to $(NumRows)-1
			LET vunits_id_inner=Peek('units_id',$(j),'temp_actions');
        	IF vunits_id = vunits_id_inner then        	
    			LET vgame_time=Peek('game_time',$(j),'temp_actions');
				LET vlat=Peek('latitude',$(j),'temp_actions');
                LET vlon=Peek('longitude',$(j),'temp_actions');
            	//check to see if the game_time on this row is nearer to slider_time; iteratively should end up with time brackets
            	IF vgame_time >= $(vslider_time) and vgame_time <= vmaxtime then
            		LET vmaxtime = vgame_time;
                    LET vlatmax = vlat;
                    LET vlonmax = vlon;
            	END IF;
          		IF vgame_time <= $(vslider_time) and vgame_time >= vmintime then
            		LET vmintime = vgame_time;
                    LET vlatmin = vlat;
                    LET vlonmin = vlon;
            	END IF;
        		// Warning: This is a STRAIGHT interpolation. Replace this with a Vincenty algorithm to make it accurate over long distances!
            	LET iterated_lat = ((vslider_time - vmintime) / (vmaxtime - vmintime)) * (vlatmax - vlatmin) + vlatmin;
            	LET iterated_lon = ((vslider_time - vmintime) / (vmaxtime - vmintime)) * (vlonmax - vlonmin) + vlonmin;
            END IF;
        NEXT j; 
        [iterated_position]:
    	ADD ONLY LOAD *
        Inline [
'units_id','iterated_position.lat','iterated_position.lon'
$(vunits_id), $(iterated_lat), $(iterated_lon)];
		//we should now have iterated through the table only finding ONE unit with time bracketed between the two nearest points. (These points might be vmaxtime or vmintime)	
	NEXT i;
    If IsPartialReload() then 
    	STORE iterated_position into $(vQVDPath)iterated_position.qvd (qvd);
    End If
end sub

5) And finally, in the visualization I use the interpolated positions in my map, and they can move. It still isn't quote as fast as I would like but a good partial solution.

daturpin
Partner - Creator II
Partner - Creator II
Author

Oh, and a last thing. For this to work set the Vlizlib to "partial app reload"

And it is important to determine what tables are updated in a partial vs. full reload!  Took me a day to figure that part out! Here is what I learned:

Statement Full Reload Partial Reload
LOAD YES NO
ADD/REPLACE/MERGE LOAD YES YES
ADD/REPLACE/MERGE ONLY LOAD NO YES