Extracting field transformation from app load script [Qlik Sense]

[budvesota]

Is there a way, using the API, to retrieve the transformations applied to a field during the data load script?

For example:

MyTable:

LOAD num(mystringfield) as MyNumField

RESIDENT tmp_table;

I'd like to retrieve the final field name (MyNumField) as well as the transformation applied (num(mystringfield)).

I know some tools like Nodegraph do this kind of task (lineage) but we're not big enough to warrant such a tool and was also interested in trying it myself. I have been unable to find this information so far.

[AustinSpivey]

I don't believe there are any Qlik JSON or QRS APIs that you'd be able to leverage to truly accomplish something like this. As far as I know and based on some research for this question, it doesn't really seem like there'd be a solid way of achieving this without actually creating a lot of your own tooling. I think it'd be a pretty big lift.

The problem is (as far as I can tell) that the Qlik Engine does not expose any part of the parsing/linting/planning/execution path of an app's load script except for the bare-bones stuff returned from the GetProgress() method. And then none of that "inter-load" info is stored or associated to the resulting tables and fields once a reload is complete except for the the source table, which is not enough to go off of for any real lineage analysis.

So it seems to me that in order to actually accomplish a full, in-depth data model lineage, you'd have to essentially write your own Qlik script parser and, probably, your own Qlik Engine.

As far as the parsing goes, you could maybe achieve this using the global GetBaseBNFString() method to get the BNF grammar for Qlik script, then use a tool like pyparse or ANTLR to create a Qlik script parser.

To be honest, I'm not totally sure where you'd go from there but I guess you would presumably use the tokenized results of the parsed script to "work backwards" up the tree to figure out the providence of any particular field.