topic Re: Parse text (RegEx ParseAll ?) in Qlik Automate

Parse text (RegEx ParseAll ?)

xyz_1011 — Thu, 13 Apr 2023 15:32:07 GMT

Hi all,

i have a regular Qlik script file. Among other script content, the script also includes multiple $(Must_Includes) statements.

Within one of my automations, i am receiving this file and i managed to get to its content (basically a long string). From this string I want to

extract all occurances of include statements in a given file and and create a list of include statements - i.e., like this: $(Must_Include=$(vModulesRoot)SALES\Revenue.qvs);, $(Must_Include=$(vModulesRoot)SALES\Customer.qvs);
Would i be able to do this with a regex() ? Or any other way in an automation ?
Once i have extracted all includes and build a list of them, i would like to loop over this list of include statements and
1. get the filename (following the above example: Revenue.qvs and Customer.qvs) of every include statement and
2. next to it: also the directory (following the above example: SALES)

Any advise on how to get there wohld be greatly apreciated!

Thanks in advance!

Re: Parse text (RegEx ParseAll ?)

AfeefaTk — Fri, 14 Apr 2023 07:01:35 GMT

Hi @xyz_1011

Let me try to understand the workflow.

So you are getting the script content using the Get Load Script block and want to get all the include statements from the script, add them to the list and loop over this list, get the filename and folder name from each item.

Do you have the automation on which you are working?

Thanks

Re: Parse text (RegEx ParseAll ?)

xyz_1011 — Fri, 14 Apr 2023 11:47:04 GMT

Hey @AfeefaTk - i believe outlining the whole use case would be something for a dedicated call (which I'd be happy to jump on if you like 🙂 ) rather than this forum post. What i am after is part of a complex use case. But let me try to illustrate with the attached (simplified) example automation as well as the following example description:

I have a script (QVS file which i am retreiving from a git lab repo). The script looks something like this (might vary from script to script):

$(Must_Include=$(vModulesRoot)03_SALES\MOD_Revenue.qvs);
call StoreIntoSDL ('MOD', 'Revenue', '03_SALES', 'Revenue');
call CreateMetaData ('Revenue', 'GEN_MOD_Revenue.qvs', 'MOD_Revenue.qvs', '03_SALES');

//=========================================================================================
// Trace ********* started...
//=========================================================================================
$(Must_Include=$(vModulesRoot)04_MARKETING\MOD_Campaign.qvs);
call StoreIntoSDL ('MOD', 'Campaign', '04_MARKETING', 'Campaign');
call CreateMetaData ('Campaign', 'GEN_MOD_Campaign.qvs', 'MOD_Campaign.qvs', '04_MARKETING');
I base decode the QVS file to get the content as a string
From this string i want to extract all include statements. To be specific, i want to create another string (iedeally a list) with every extracted include statement, which looks like this (every item represents a path in a gitlab repo):

04_MARKETING\MOD_Campaign.qvs
03_SALES\MOD_Revenue.qvs
After that i want to iterate over every item i built in the previous step and use a GitLab Get File From Repository to fetch the file...

I hope, what i am after became a bit clearer...in essence, what i am struggling with is: Extract the substring from every include and then iterate over every extraction.

Thanks a lot upfron!

Re: Parse text (RegEx ParseAll ?)

blaise — Fri, 14 Apr 2023 12:08:51 GMT

Im far from a regexp expert but rather good a googling 🙂 so what about apply a regExp parse all function on the string:

(?<=Must_Include=)(.*)(?=;)

I tested it with some examples in a variable (vString) with the output block with the following code (formula parsing off):

{regexparseall: {$.vString}, '(?<=Must_Include=)(.*)(?=;)'}

The output will not include Must_Include and the semi-colon, the regExp takes the string between these two strings. It also includes a ) at the end as the regexp fails if I use ); instead of ; as the end-string.

Re: Parse text (RegEx ParseAll ?)

AfeefaTk — Fri, 14 Apr 2023 12:38:48 GMT

Hi @xyz_1011

This formula works

{regexparseall: {$.vString}, '(?<=Must_Include=)(.*)(?=;)'}

But you also need to remove this $(vModulesRoot) from the path right?

Re: Parse text (RegEx ParseAll ?)

AfeefaTk — Fri, 14 Apr 2023 13:35:34 GMT

Hi @xyz_1011

I have tried this using custom code block.

Please take a look at the json file and let me know if that works for you.

Thanks

Re: Parse text (RegEx ParseAll ?)

AfeefaTk — Fri, 14 Apr 2023 13:45:58 GMT

Hi @xyz_1011

I have also done this using regex parse all formula.

Hope this works.

Thanks

Re: Parse text (RegEx ParseAll ?)

xyz_1011 — Fri, 14 Apr 2023 14:35:56 GMT

Hey @AfeefaTk Very elegant! I had no clue that there is a way to work with PHP. This is cool!

(obviously, it also does the job 😉 Thanks!!)

Re: Parse text (RegEx ParseAll ?)

AfeefaTk — Fri, 14 Apr 2023 14:48:52 GMT

Hi @xyz_1011

I would suggest you try my recent automation which uses the regex formula by replacing the custom code block. This should be a more clean solution than the custom code one.

Thanks

Re: Parse text (RegEx ParseAll ?)

xyz_1011 — Mon, 17 Apr 2023 07:33:50 GMT

Hey @AfeefaTk thanks again for your help. I am running into an issue though, when trying to get a file from my repo after parsing the includes:

1. storing every extracted & parsed include ina variable:

2. trying to get a file from my repo using this variable:

3. Runn ing into this error, where the / seems to cannot be decoded properly:

Any idea how to solve this ?

Re: Parse text (RegEx ParseAll ?)

xyz_1011 — Mon, 17 Apr 2023 08:59:15 GMT

Got it done myself. Thanks.

Re: Parse text (RegEx ParseAll ?)

AfeefaTk — Mon, 17 Apr 2023 09:47:51 GMT

Hi @xyz_1011

Based on your recent query:

Regarding to the accepted solution in my post, i realized, that ideally i not only need the includes() extracted and parsed, but also every call StoreIntoSDL() line following the include. Is it possible that the custom code segment gives me two variables:

1. the extracted and parsed include()

2. the call StoreIntoSDL line that comes after ?

Meaning, given this script piece:

$(Must_Include=$(vModulesRoot)03_SALES\MOD_Revenue.qvs);
call StoreIntoSDL ('MOD', 'Revenue', '03_SALES', 'Revenue');

I would like to get the following items back from the custom code block:

vInclude = 30_SALES\MOD_Revenue.qvs (<-- that works already in your solution)

vStore = call StoreIntoSDL ('MOD', 'Revenue', '03_SALES', 'Revenue'); (<-- this would need to come on top)

I am attaching automation json for your reference.

Hope this works!!

Thanks

Re: Parse text (RegEx ParseAll ?)

xyz_1011 — Mon, 17 Apr 2023 10:05:10 GMT

Thanks, @AfeefaTk - this does al,most what i need. However, i am not able to process the output of each iteration together. Meaning: In the first loop i expect to receive the first parsed include() and the first extracted call() then i do something with this output. In the second iteration the secon parsed include() together with the second call(), then i do something with this output ... Hope, this makes sense. Thanks again!

Re: Parse text (RegEx ParseAll ?)

AfeefaTk — Mon, 17 Apr 2023 10:57:49 GMT

Hi @xyz_1011

Please find the attached automation

Thanks