Schematron validation using XSLT using BizTalk – Part 3

This post is a continuation to the Schematron validation using XSLT in BizTalk blog series

The source code for this blog post has been uploaded in the below location

http://code.msdn.microsoft.com/Schematron-validation-b4d892ee
Now that you know about the basic schematron XSLT implementation, we will now see how we can achieve this in BizTalk.

High Level steps for BizTalk implementation
1. Create the schematron-output schema which we discussed in the last section
2. Create a custom pipeline component( Any Category type, so that we can have the component in any of the pipeline stage, except disassembling stage)
a. Implement Custom UI functionality so that we can have the Schematron rules as property. We are also going to add a property to hold the physical path to the conformance1-5.xsl. So it is important that the conformance1-5.xsl and the skeleton1-5.xsl is placed in appropriate directories.
b. If we have any schematron errors, then we will promote the message type with the schematron output message type.
3. Apply an inbound map in the receive port if transformation is required(Optional Step)

Detailed Steps for BizTalk implementation
1. I will skip the creation of schematron-output schema .
2. Custom Pipeline component

The pipeline component has three parts
a. The component itself
b. Custom UI form
c. Custom UI Type Editor class

The custom UI is a windows form which can accept the XSLT and store it in a XML serialized format. This is pretty useful, as we can load the XSLT stylesheet from the pipeline component property itself.

The code for Custom UI and Type Editor Class are self-explanatory and I will proceed with the actual transformation logic.

public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inMsg)
{
if (Enabled)
{
try
{
ReadOnlySeekableStream seekableStm = new ReadOnlySeekableStream(inMsg.BodyPart.Data);

VirtualStream vs = null;

XmlTextReader stylesheet = new XmlTextReader(new StringReader(this.SchematronXSLT));

XmlTextReader conformanceStylesheet = new XmlTextReader(this.ConformanceXSLTPath);

// Load transform
BTSXslTransform trans = new BTSXslTransform();
trans.Load(conformanceStylesheet);

XmlTextReader inputReader = new XmlTextReader(seekableStm);

//Create memory stream to hold transformed data.
vs = new VirtualStream(VirtualStream.MemoryFlag.AutoOverFlowToDisk);

//Preform transform
trans.ScalableTransform(stylesheet, null, vs, new XmlUrlResolver(), false);

vs.Seek(0, SeekOrigin.Begin);

XmlTextReader validatingXSLReader = new XmlTextReader(vs);
BTSXslTransform ValidatingTrans = new BTSXslTransform();
ValidatingTrans.Load(validatingXSLReader);

VirtualStream vs1 = new VirtualStream(VirtualStream.MemoryFlag.AutoOverFlowToDisk);

ValidatingTrans.ScalableTransform(inputReader, null, vs1, new XmlUrlResolver(), false);
vs1.Seek(0, SeekOrigin.Begin);
XmlDocument outputDoc = new XmlDocument();
outputDoc.Load(vs1);
if (outputDoc.SelectNodes(“//failed-assert”).Count > 0)
{
vs.Flush();
vs1.Flush();
vs1.Seek(0, SeekOrigin.Begin);
inMsg.BodyPart.Data = vs1;
inMsg.Context.Promote(“MessageType”, “http://schemas.microsoft.com/BizTalk/2003/system-properties”, “http://www.ascc.net/xml/schematron#schematron-output”);
inMsg.BodyPart.Data.Position = 0;
}
else
{
seekableStm.Seek(0, SeekOrigin.Begin);
inMsg.BodyPart.Data = seekableStm;
}

}
catch (Exception ex)
{
System.Diagnostics.EventLog.WriteEntry(this.Name, ex.Message);
}
}

return inMsg;

}

Key things from the above code is I have used BTSXslTransform class to do the transformation and used VirtualStream to hold the transformed data. One nice feature about VirtualStream is that it is initialized with VirtualStream.MemoryFlag.AutoOverFlowToDisk i.e. when the internal memory is not sufficient to hold the transformed data, it will be stored to the disk. This is greatly useful in case of large messages.

Other than that, the code is quite simple to follow.

Summary

So far, we have seen the development of the schematron XSLT infrastructure and how to use it with BizTalk. This part of the tutorial is only a beginner and we will see more complex schematron in the upcoming posts.

Schematron validation using XSLT in BizTalk – Part 2

This post is a continuation to the Schematron validation using XSLT in BizTalk blog series

The source code for this blog post has been uploaded in the below location

http://code.msdn.microsoft.com/Schematron-validation-b4d892ee

So, How to do schematron validation using XSLT in BizTalk?

High level steps
1. Define the schematron rules in the .sch file
2. Use XSLT processor(MSXSL) and pass the Rules.sch file and the Conformance1-5.xsl file
3. STEP 2 would produce a Validating XSLT file, let’s call Valdating.xsl.
4. Use XSLT processor and pass the XML document and Validating.xsl. This would produce the final schematron results.

Pros of using this approach
1. There is no dependency with any third party library
2. The schematron skeleton XSLT template that we use is ISO standard, open source and is used widely in various companies.
3. Complete use of XSLT functions like document(), key() etc.

Cons of using this approach
1. Compared to the XPATH implementation it is slower, as there is an additional step of preparing the validating XSLT.
2. The initial setup is a bit complex, but when you have setup the Pipeline components and required schemas and Schematron Skeleton XSLTs, then it is easier.

Detailed Implementation
We have seen the high level steps and now it’s time to get down deep into the implementation details.
Before going into the implementation, let’s take a scenario, so that we can use that for our implementation
A manufacturing company has adopted an e-Commerce solution by integrating its services with its client. Its clients can send purchase order by placing the files in a FILE folder. The Manufacturing Company is responsible for validating the structure of the message before processing the Purchase Order.
We need to have a consistent approach for validating the semantics of the schema and report any errors. We will go step in step in implementing this solution.

1. Define the PurchaseOrder schema

Below is the high level structure of the schema.

Please also check the schema file from the below link.

http://code.msdn.microsoft.com/Schematron-validation-b4d892ee

Okay, we have defined the schema. The next step would be writing schematron rules.

2. Define schematron rules

To start with, we will define quite simple schematron rules.

Schematron schema has a predefined structure. Below is the skeleton structure of the schematron.

<?xml version=”1.0″ encoding=”UTF-8″?>

<sch:schema xmlns:sch=”http://www.ascc.net/xml/schematron&#8221; xmlns:xs=”http://www.w3.org/2001/XMLSchema&#8221; queryBinding=”xslt2″

</sch:schema>
We will add the namespace to it.

<sch:ns uri=” http://www.nmstech.uk/ecommerce/schemas/v01 ” prefix=”nms” />

And we will add a very simple rule

<sch:pattern name=”Check Existence”>

<sch:rule context=”nms:PurchaseOrder”>

<sch:assert test=”nms:Id”>PurchaseOrder Id is missing or cannot be empty</sch:assert>

</sch:rule>

</sch:pattern>
Let’s put it together.

<?xml version=”1.0″ encoding=”UTF-8″?>

<sch:schema xmlns:sch=”http://www.ascc.net/xml/schematron&#8221; xmlns:xs=”http://www.w3.org/2001/XMLSchema&#8221; queryBinding=”xslt2″ >

<sch:ns uri=” http://www.nmstech.uk/ecommerce/schemas/v01 ” prefix=”nms” />

<sch:pattern name=”Check Existence”>

<sch:rule context=”nms:PurchaseOrder”>

<sch:assert test=”nms:Id”>PurchaseOrder Id is missing</sch:assert>

</sch:rule>

</sch:pattern>

</sch:schema>
That’s it; we have got a very simple schematron schema.

3. Define schema to populate error information
When we run the XSLT processor with conformance1-5.xslt, we will get a schematron-output message with all the error information. We will use the same structure to define a schema, so that we can apply an inbound map to transform this into a more meaningful message or transform this to a client response message.

4. Download Schematron Skeleton XSLTs

The conformance1-5.xsl and the skeleton1-5.xsl has been provided by Academia Sinica Computing Center, Taiwan

As already said, the above XSLTs are open source and can be downloaded from

http://xml.ascc.net/schematron/1.5/skeleton1-5.xsl
http://xml.ascc.net/schematron/1.5/conformance1-5.xsl

5. Download MSXSL 3.0 for testing purpose
Download the command line utility MSXSL 3.0 for testing purpose. In the actual development we will use the BTSXslTransform class to apply the XSLTs.

6. Generate Validating XSLT using XSLT processor and Skeleton XSLTs

Execute the below to generate the validating XSLT

msxsl -o PO-Validating.xsl PurchaseOrder.sch conformance1-5.xsl

Below is the auto generated Validating XSLT

​
​

<?xml version=”1.0″ encoding=”UTF-16″ standalone=”yes”?>

<xsl:stylesheet version=”1.0″ nms:dummy-for-xmlns=”” xmlns:xsl=”http://www.w3.org/1999/XSL/Transform&#8221; xmlns:sch=”http://www.ascc.net/xml/schematron&#8221; xmlns:nms=” http://www.nmstech.uk/ecommerce/schemas/v01 “>

<xsl:output method=”xml” omit-xml-declaration=”no” standalone=”yes” indent=”yes” />

<xsl:template match=”*|@*” mode=”schematron-get-full-path”>

<xsl:apply-templates select=”parent::*” mode=”schematron-get-full-path” />

<xsl:text>/</xsl:text>

<xsl:if test=”count(. | ../@*) = count(../@*)”>@</xsl:if>

<xsl:value-of select=”name()” />

<xsl:text>[</xsl:text>

<xsl:value-of select=”1+count(preceding-sibling::*[name()=name(current())])” />

<xsl:text>]</xsl:text>

</xsl:template>

<xsl:template match=”/”>

<schematron-output title=”” schemaVersion=”” phase=”#ALL”>

<ns uri=” http://www.nmstech.uk/ecommerce/schemas/v01 ” prefix=”nms” />

<active-pattern name=”Check Existence”>

<xsl:apply-templates />

</active-pattern>

<xsl:apply-templates select=”/” mode=”M1″ />

</schematron-output>

</xsl:template>

<xsl:template match=”nms:PurchaseOrder” priority=”4000″ mode=”M1″>

<fired-rule id=”” context=”nms:PurchaseOrder” role=”” />

<xsl:choose>

<xsl:when test=”nms:Id” />

<xsl:otherwise>

<failed-assert id=”” test=”nms:Id” role=””>

<xsl:attribute name=”location”>

<xsl:apply-templates select=”.” mode=”schematron-get-full-path” />

</xsl:attribute>

<text>PurchaseOrder Id is missing</text>

</failed-assert>

</xsl:otherwise>

</xsl:choose>

<xsl:apply-templates mode=”M1″ />

</xsl:template>

<xsl:template match=”text()” priority=”-1″ mode=”M1″ />

<xsl:template match=”text()” priority=”-1″ />

</xsl:stylesheet>​

Below is a sample PurchaseOrder instance




​

<?xml version=”1.0″ encoding=”UTF-8″?>

<!–Sample XML file generated by XMLSpy v2011 (http://www.altova.com)–>

<nms:PurchaseOrder xsi:schemaLocation=”http://www.nmstech.uk/ecommerce/schemas/v01 PurchaseOrder.xsd” xmlns:nms=”http://www.nmstech.uk/ecommerce/schemas/v01&#8243; xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”&gt;

<nms:Timestamp>2001-12-17T09:30:47Z</nms:Timestamp>

<nms:POHeader>

<nms:SoldTo>

<nms:PartyId>TestParty</nms:PartyId>

<nms:Address/>

</nms:SoldTo>

<nms:BillTo>

<nms:PartyId>TestParty</nms:PartyId>

<nms:Address/>

</nms:BillTo>

<nms:BuyerOrderId>TstBuyer</nms:BuyerOrderId>

<nms:Supplier>

<nms:PartyId>TestParty</nms:PartyId>

<nms:Address/>

</nms:Supplier>

<nms:TermsCode>TestTerms</nms:TermsCode>

<nms:TranspCode>TestTranps</nms:TranspCode>

<nms:DateOrdered>1967-08-13</nms:DateOrdered>

<nms:Backorder>Y</nms:Backorder>

<nms:PORevisionNumber>0</nms:PORevisionNumber>

<nms:POStatusIndicator>N</nms:POStatusIndicator>

<nms:ASNRequirement>Y</nms:ASNRequirement>

<nms:POFileType>P</nms:POFileType>

<nms:ShipTo>

<nms:PartyId>TestParty</nms:PartyId>

<nms:Address/>

</nms:ShipTo>

</nms:POHeader>

<nms:PODetails>

<nms:Items>

<nms:Item>

<nms:POLineNbr>0</nms:POLineNbr>

<nms:BuyerItemId>TestBuyer</nms:BuyerItemId>

<nms:Qty>0.0</nms:Qty>

<nms:QtyUOM>aaaaa</nms:QtyUOM>

<nms:UCValue>0.0</nms:UCValue>

<nms:UCCurrencyCode>aaa</nms:UCCurrencyCode>

<nms:BarCodeId>aaaaaaaaaaaaaa</nms:BarCodeId>

<nms:BarCodeType>GTIN-13</nms:BarCodeType>

</nms:Item>

</nms:Items>

</nms:PODetails>

</nms:PurchaseOrder>

If you inspect the xml message, you can see that the ID field is missing. We will now run the XSLT processor against this xml instance and the Validating XSLT

​Below is the command

msxsl -o PO-Result.xml PO-Test.xml PO-Validating.xsl

And below is the contents of PO-Result.xml

​
​

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?>

<schematron-output title=”” schemaVersion=”” phase=”#ALL” xmlns:sch=”http://www.ascc.net/xml/schematron&#8221; xmlns:nms=”http://www.nmstech.uk/ecommerce/schemas/v01″&gt;

<ns uri=” http://www.nmstech.uk/ecommerce/schemas/v01 ” prefix=”nms” />

<active-pattern name=”Check Existence”/>

<fired-rule id=”” context=”nms:PurchaseOrder” role=”” />

<failed-assert id=”” test=”nms:Id” role=”” location=”/nms:PurchaseOrder[1]”>

<text>PurchaseOrder Id is missing</text>

</failed-assert>

</schematron-output>

What we have done and achieved so far

1. Defined a schematron rules
2. Defined a Schematron Output schema
3. Executed the XSLT processor and got the results.

If you want to have schematron functionality for another flow, all you have to do is create the step 1, i.e. define schematron rules and that’s it, the rest all are the same.

Schematron Validation using XSLT in BizTalk – Part 1

This blog post is a 3 part series.

The source code for this blog post has been uploaded in the below location

http://code.msdn.microsoft.com/Schematron-validation-b4d892ee

BizTalk has an out of the box XML validation pipeline component to validate the semantics of the XSD schema and return the error message. However it has a lot of drawbacks
– can’t have custom error messages for failures
– can’t have all the error messages at one shot
Enter Schematron. Schematron is a language for making assertions about patterns found in XML documents. It is a rule based validation and is an ISO/IEC Standard.
There are currently many ways for implementing schematron. If we talk about BizTalk and .Net, there are two ways.
1. XPATH based implementation
2. XSLT based implementation

XPATH based in .Net Framework

Daniel Cazzulino, Microsoft XML MVP, designed a class library called Schematron.Net which provides classes for validating XML documents against the schematron schemas. This is done by the following
1. Embed Schematron rule patterns in BizTalk XSD schemas.
2. Write a custom pipeline component to call the Schematron.Net assembly to do the validation or use the Schematron Pipeline component available in Codeplex.

The above approach has some issues
1. There is a dependency to this class library. Although this library is open source, when you build BizTalk applications for enterprises, management doesn’t really like the idea of having 3rd Software libraries in the code. In case there are issues with the library, there will be no one to support.
2. A lot of XSLT functions are not available in this implementation

XSLT based Implementation

XSLT based Implementation works in two steps (Actually its 4 steps, however the extra 2 steps are actually required in complex scenarios)
1. The Schematron schema (.sch) is first turned into a validating XSLT stylesheet by transforming it with an XSLT stylesheet provided by Academica Sinica Computing Centre. These stylesheets (schematron-basic.xsl, schematron-message.xsl, schematron-report.xsl and conformance1-5.xsl) can be found at the Schematron site and the different stylesheets generate different output.
2. This validating stylesheet is then used on the XML instance document and the result will be a report that is based on the rules and assertions in the original Schematron schema.