How To Remove Constraints During a Dataflow

    Constraints are often set on Composite Type attributes to cleanse or filter data as it is read from a source, such as an RDBMS. Once the cleansing or filtering is done, the constraints are no longer needed. Nevertheless, the constraints defined for the Composite Type attributes continue to be applied as the attributes are propagated through the dataflow. This requires extra processing and thus can slow performance of the application. For that reason, you might want to remove the constraints from Composite Type attributes used downstream in the dataflow.

    There are other reasons as well for removing constraints in the middle of a dataflow. You might want to apply a different constraint downstream, as when processing results in a string length would fail the original constraint. Or you might need to reprocess the records that failed the constraints initially. To process the reject records on another pass, you would have to apply a Composite Type whose attributes do not have the constraints the records failed to pass.

    In addition, if you wanted to recombine the rejected records with those that passed the constraint validation, you will need to define a set of attributes that does not have the constraints. Otherwise, the records that have been added back will fail again.

    The simplest way to change constraints on a set of Composite Type attributes is to pass the data through a Transform operator and change the output attributes.

    1. Add a Transform operator downstream of the operator emitting the constrained Composite Type attributes.
      For example, the following dataflow has a Transform operator after a Read Table operator that has attribute constraints and a Write File operator connected to its reject port.
    2. Select the Transform operator and click the Edit Rules button in the Properties panel.
      The output attributes are identical to the input attributes and so have the constraints defined for them. Because of that, the existing output attributes cannot be changed. The arrows displayed to the left of the output attributes indicate that the output attributes have been taken from the input attributes.
    3. Open the Table Schema used by the Read Table operator.
      The Composite Type mapped to the Schema is a Local Type here.
      Do not change the constraints on the Local Type attributes because those are the constraints used in the Read Table operator.
    4. Select the Composite Type option from the Share button drop-down menu.
      This new Shared Composite Type will be used to create another Composite Type whose attributes you can change. That Type's attributes will then be used to replace the Transform operator's output attributes.
    5. Name the new Composite Type.
      The new Type now appears in the Types folder in the Explorer panel.
    6. Right-click on the new Composite Type in the Explorer panel and select the Make a Copy option from the menu.
    7. Rename the new Composite Type with an appropriate name.
    8. Open the new Composite Type.
    9. Edit each attribute to remove the constraints.
      Of course, you can change the constraints rather than remove them if subsequent processing requires that the data be constrained in other ways.
    10. Go back to the Rules Editor and import the attributes from the new Shared Composite Type.
      The newly imported attributes will replace the existing attributes, and the new attributes will display a diamond to the left instead of the arrow originally displayed.
    11. Select each input attriubte and drag a line from it to the corresponding output attribute.
      Configuration of the Transform operator is now complete. The attriubtes from the new Composite Type, on which you changed the consstraints in Step 9, will be emitted downstream by the Transform operator.