Help with Multi-Level XML Filtering

[MAnywar]

Dear all, I have an issue that I have failed to get a solution to. and it is to do with multi level scenarios.

1. 1st if a message contains ZOP

Then it should be dropped <ZOP> tag, then it should be ignored/dropped and the next message is read: sample message below.

XML Tag:

<BAR_P12 xmlns="urn:hl7-org:v2xml">

  <MSH>

       <MSH.16>NE</MSH.16>

    <MSH.18>8859/1</MSH.18>

  </MSH>

<PID>

    <PID.1>1</PID.1>

    <PID.2>

      <CX.1>61457074</CX.1>

    </PID.2>

     <PID.7>

      <TS.1>19560979</TS.1>

    </PID.7>

    </PID>

  <ZOP>

    <ZOP.1>19272</ZOP.1>

    <ZOP.13>

      <UNKNOWN.1>HHHH</UNKNOWN.1>

      <UNKNOWN.2>YYYYY</UNKNOWN.2>

      <UNKNOWN.3>Claudius</UNKNOWN.3>

      <UNKNOWN.6>SSSSS</UNKNOWN.6>

    </ZOP.13>

    <ZOP.13>

       <UNKNOWN.1>HHHH</UNKNOWN.1>

      <UNKNOWN.2>YYYYY</UNKNOWN.2>

      <UNKNOWN.3>Claudius</UNKNOWN.3>

      <UNKNOWN.6>SSSSS</UNKNOWN.6>

    </ZOP.13>

  </ZOP>

</BAR_P12>

1. ​2. Also if a message has the xml format below; the ZOP tag should be filtered out and only the the DGN part is read, But this is almost like achieved:

<BAR_P12 xmlns="urn:hl7-org:v2xml">

  <MSH>

       <MSH.16>NE</MSH.16>

    <MSH.18>8859/1</MSH.18>

  </MSH>

<PID>

    <PID.1>1</PID.1>

    <PID.2>

      <CX.1>61457074</CX.1>

    </PID.2>

     <PID.7>

      <TS.1>19560979</TS.1>

    </PID.7>

    </PID>

  <ZOP>

    <ZOP.1>19272</ZOP.1>

    <ZOP.13>

      <UNKNOWN.1>HHHH</UNKNOWN.1>

      <UNKNOWN.2>YYYYY</UNKNOWN.2>

      <UNKNOWN.3>Claudius</UNKNOWN.3>

      <UNKNOWN.6>SSSSS</UNKNOWN.6>

    </ZOP.13>

    <ZOP.13>

       <UNKNOWN.1>HHHH</UNKNOWN.1>

      <UNKNOWN.2>YYYYY</UNKNOWN.2>

      <UNKNOWN.3>Claudius</UNKNOWN.3>

      <UNKNOWN.6>SSSSS</UNKNOWN.6>

    </ZOP.13>

  </ZOP>

  <DG1>

    <DG1.1>1</DG1.1>

    <DG1.5>

      <TS.1>20221012112900</TS.1>

    </DG1.5>

    <DG1.6>EL</DG1.6>

    <DG1.15>2</DG1.15>

    <DG1.16>

      <XCN.1>AKKERMANN-H</XCN.1>

      <XCN.23>

        <CWE.1>LSHNO</CWE.1>

      </XCN.23>

    </DG1.16>

    <DG1.20>

      <EI.1>85759256</EI.1>

    </DG1.20>

    <DG1.21>A</DG1.21>

  </DG1>

</BAR_P12>

1. 3. The other scenario is; if xml <DG1.21> tag has the value  <DG1.21>FILTER THIS</DG1.21>. as seen below, it should also not be processed. How can one achieve this ?

<BAR_P12 xmlns="urn:hl7-org:v2xml">

  <MSH>

       <MSH.16>NE</MSH.16>

    <MSH.18>8859/1</MSH.18>

  </MSH>

<PID>

    <PID.1>1</PID.1>

    <PID.2>

      <CX.1>61457074</CX.1>

    </PID.2>

     <PID.7>

      <TS.1>19560979</TS.1>

    </PID.7>

    </PID>

  <DG1>

    <DG1.1>1</DG1.1>

    <DG1.5>

      <TS.1>20221012112900</TS.1>

    </DG1.5>

    <DG1.6>EL</DG1.6>

    <DG1.15>2</DG1.15>

    <DG1.20>

      <EI.1>85759256</EI.1>

    </DG1.20>

    <DG1.21>FILTER THIS</DG1.21>

  </DG1>

</BAR_P12>

Thank you all. for the help.

Here is a screenshot of what I have so far: But unfortunately this is not working as the lower components are throwing an exception error that atHashInput needs to be declared before reading from it.

Any help is appreciated.

[Anonymous]

Hi

tHashOutput has not been initialised before you read data from it using tHashInput, a nice trick is to set up your tHashOutputs with a dummy initialisation at the beginning of your Job. Simply set up a tFixedFlowInput with the schema you expect, set the number of rows to 0 and connect it to a tHashOutput that you will append to. This way, the tHashOutput has always been initialised and your job won’t crash if the first time (of multiple times) you want to append to it, there is no data.

About your scenario, the XML string might have the different structure? If so, I think you need to use different tExractXMLField components to extract data from each structure.

Regards

Shong

[MAnywar]

Hi @Shicong Hong​ 

Thanks for your reply. I had done this with number of rows set to 1. Now I changed to number of rows 0, but still the error and job breaking is still occurring. I am reading two test xml files; 1 having scenario 1 which requires it to just be discarded, and then the next file which has scenario two (containing data that is fits the schema and some that do not fit the schema).

And when I change the Number of Rows to 0 in the tFixedFlowInput, I do not get any output from the tLogs.

tHashInput 7 reads from thashOutput 3

tHashInput 8 reads from thashOutput 4

tHashInput 9 reads from thashOutput 5

[MAnywar]

I have tried to make some adjustments;

But now i feel like I have another issue:

I am reading from a pipeline that at times send me data that I need to disregard; at times I have to read and process (Filter)

But now I have an issue where by I have to create a join to form one xml document based on the data in the tHash output. However, at times tHashOutput 6 contains null as the data going in it was filtered out (not needed). This causes my job to keep breaking due to the fact that The hash 1 keeps complaining that it is not initialized. It is really getting frustrating. I have tried most possible options including redesign, but and all filtering, but none seems to work for me.

Please if you can advise me on how to go about this, I will surely be grateful.

tMap1 has some formatings it is doing on both arms and then tmap2 is joing the two. But tHashInput_2 always has data. it is input1 that at times contains nulls.

[Anonymous]

If you have several tHashOutput components in the job, go to check and make sure tHashInput linked to the right tHashOutput component. Can you share a screenshot of your current Job design?

Regards

Shong​

[MAnywar]

Unfortunately I could not share any screenshot, but managed to work around. "Clear cache after reading" and "Append" options to do away with the errors.

Thanks.