Schematron validation using XSLT in BizTalk – Part 2

This post is a continuation to the Schematron validation using XSLT in BizTalk blog series

The source code for this blog post has been uploaded in the below location

http://code.msdn.microsoft.com/Schematron-validation-b4d892ee

So, How to do schematron validation using XSLT in BizTalk?

High level steps
1. Define the schematron rules in the .sch file
2. Use XSLT processor(MSXSL) and pass the Rules.sch file and the Conformance1-5.xsl file
3. STEP 2 would produce a Validating XSLT file, let’s call Valdating.xsl.
4. Use XSLT processor and pass the XML document and Validating.xsl. This would produce the final schematron results.

Pros of using this approach
1. There is no dependency with any third party library
2. The schematron skeleton XSLT template that we use is ISO standard, open source and is used widely in various companies.
3. Complete use of XSLT functions like document(), key() etc.

Cons of using this approach
1. Compared to the XPATH implementation it is slower, as there is an additional step of preparing the validating XSLT.
2. The initial setup is a bit complex, but when you have setup the Pipeline components and required schemas and Schematron Skeleton XSLTs, then it is easier.

Detailed Implementation
We have seen the high level steps and now it’s time to get down deep into the implementation details.
Before going into the implementation, let’s take a scenario, so that we can use that for our implementation
A manufacturing company has adopted an e-Commerce solution by integrating its services with its client. Its clients can send purchase order by placing the files in a FILE folder. The Manufacturing Company is responsible for validating the structure of the message before processing the Purchase Order.
We need to have a consistent approach for validating the semantics of the schema and report any errors. We will go step in step in implementing this solution.

1. Define the PurchaseOrder schema

Below is the high level structure of the schema.

PurchaseOrder

Please also check the schema file from the below link.

http://code.msdn.microsoft.com/Schematron-validation-b4d892ee

Okay, we have defined the schema. The next step would be writing schematron rules.

2. Define schematron rules

To start with, we will define quite simple schematron rules.

Schematron schema has a predefined structure. Below is the skeleton structure of the schematron.

<?xml version=”1.0″ encoding=”UTF-8″?>

<sch:schema xmlns:sch=”http://www.ascc.net/xml/schematron&#8221; xmlns:xs=”http://www.w3.org/2001/XMLSchema&#8221; queryBinding=”xslt2″

</sch:schema>
We will add the namespace to it.

<sch:ns uri=” http://www.nmstech.uk/ecommerce/schemas/v01 ” prefix=”nms” />

And we will add a very simple rule

<sch:pattern name=”Check Existence”>

<sch:rule context=”nms:PurchaseOrder”>

<sch:assert test=”nms:Id”>PurchaseOrder Id is missing or cannot be empty</sch:assert>

</sch:rule>

</sch:pattern>
Let’s put it together.

<?xml version=”1.0″ encoding=”UTF-8″?>

<sch:schema xmlns:sch=”http://www.ascc.net/xml/schematron&#8221; xmlns:xs=”http://www.w3.org/2001/XMLSchema&#8221; queryBinding=”xslt2″ >

<sch:ns uri=” http://www.nmstech.uk/ecommerce/schemas/v01 ” prefix=”nms” />

<sch:pattern name=”Check Existence”>

<sch:rule context=”nms:PurchaseOrder”>

<sch:assert test=”nms:Id”>PurchaseOrder Id is missing</sch:assert>

</sch:rule>

</sch:pattern>

</sch:schema>
That’s it; we have got a very simple schematron schema.

3. Define schema to populate error information
When we run the XSLT processor with conformance1-5.xslt, we will get a schematron-output message with all the error information. We will use the same structure to define a schema, so that we can apply an inbound map to transform this into a more meaningful message or transform this to a client response message.

schematron-output

4. Download Schematron Skeleton XSLTs

The conformance1-5.xsl and the skeleton1-5.xsl has been provided by Academia Sinica Computing Center, Taiwan

As already said, the above XSLTs are open source and can be downloaded from

http://xml.ascc.net/schematron/1.5/skeleton1-5.xsl
http://xml.ascc.net/schematron/1.5/conformance1-5.xsl

5. Download MSXSL 3.0 for testing purpose
Download the command line utility MSXSL 3.0 for testing purpose. In the actual development we will use the BTSXslTransform class to apply the XSLTs.

6. Generate Validating XSLT using XSLT processor and Skeleton XSLTs

Execute the below to generate the validating XSLT

msxsl -o PO-Validating.xsl PurchaseOrder.sch conformance1-5.xsl

Below is the auto generated Validating XSLT


<?xml version=”1.0″ encoding=”UTF-16″ standalone=”yes”?>

<xsl:stylesheet version=”1.0″ nms:dummy-for-xmlns=”” xmlns:xsl=”http://www.w3.org/1999/XSL/Transform&#8221; xmlns:sch=”http://www.ascc.net/xml/schematron&#8221; xmlns:nms=” http://www.nmstech.uk/ecommerce/schemas/v01 “>

<xsl:output method=”xml” omit-xml-declaration=”no” standalone=”yes” indent=”yes” />

<xsl:template match=”*|@*” mode=”schematron-get-full-path”>

<xsl:apply-templates select=”parent::*” mode=”schematron-get-full-path” />

<xsl:text>/</xsl:text>

<xsl:if test=”count(. | ../@*) = count(../@*)”>@</xsl:if>

<xsl:value-of select=”name()” />

<xsl:text>[</xsl:text>

<xsl:value-of select=”1+count(preceding-sibling::*[name()=name(current())])” />

<xsl:text>]</xsl:text>

</xsl:template>

<xsl:template match=”/”>

<schematron-output title=”” schemaVersion=”” phase=”#ALL”>

<ns uri=” http://www.nmstech.uk/ecommerce/schemas/v01 ” prefix=”nms” />

<active-pattern name=”Check Existence”>

<xsl:apply-templates />

</active-pattern>

<xsl:apply-templates select=”/” mode=”M1″ />

</schematron-output>

</xsl:template>

<xsl:template match=”nms:PurchaseOrder” priority=”4000″ mode=”M1″>

<fired-rule id=”” context=”nms:PurchaseOrder” role=”” />

<xsl:choose>

<xsl:when test=”nms:Id” />

<xsl:otherwise>

<failed-assert id=”” test=”nms:Id” role=””>

<xsl:attribute name=”location”>

<xsl:apply-templates select=”.” mode=”schematron-get-full-path” />

</xsl:attribute>

<text>PurchaseOrder Id is missing</text>

</failed-assert>

</xsl:otherwise>

</xsl:choose>

<xsl:apply-templates mode=”M1″ />

</xsl:template>

<xsl:template match=”text()” priority=”-1″ mode=”M1″ />

<xsl:template match=”text()” priority=”-1″ />

</xsl:stylesheet>​

Below is a sample PurchaseOrder instance


<?xml version=”1.0″ encoding=”UTF-8″?>

<!–Sample XML file generated by XMLSpy v2011 (http://www.altova.com)–>

<nms:PurchaseOrder xsi:schemaLocation=”http://www.nmstech.uk/ecommerce/schemas/v01 PurchaseOrder.xsd” xmlns:nms=”http://www.nmstech.uk/ecommerce/schemas/v01&#8243; xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”&gt;

<nms:Timestamp>2001-12-17T09:30:47Z</nms:Timestamp>

<nms:POHeader>

<nms:SoldTo>

<nms:PartyId>TestParty</nms:PartyId>

<nms:Address/>

</nms:SoldTo>

<nms:BillTo>

<nms:PartyId>TestParty</nms:PartyId>

<nms:Address/>

</nms:BillTo>

<nms:BuyerOrderId>TstBuyer</nms:BuyerOrderId>

<nms:Supplier>

<nms:PartyId>TestParty</nms:PartyId>

<nms:Address/>

</nms:Supplier>

<nms:TermsCode>TestTerms</nms:TermsCode>

<nms:TranspCode>TestTranps</nms:TranspCode>

<nms:DateOrdered>1967-08-13</nms:DateOrdered>

<nms:Backorder>Y</nms:Backorder>

<nms:PORevisionNumber>0</nms:PORevisionNumber>

<nms:POStatusIndicator>N</nms:POStatusIndicator>

<nms:ASNRequirement>Y</nms:ASNRequirement>

<nms:POFileType>P</nms:POFileType>

<nms:ShipTo>

<nms:PartyId>TestParty</nms:PartyId>

<nms:Address/>

</nms:ShipTo>

</nms:POHeader>

<nms:PODetails>

<nms:Items>

<nms:Item>

<nms:POLineNbr>0</nms:POLineNbr>

<nms:BuyerItemId>TestBuyer</nms:BuyerItemId>

<nms:Qty>0.0</nms:Qty>

<nms:QtyUOM>aaaaa</nms:QtyUOM>

<nms:UCValue>0.0</nms:UCValue>

<nms:UCCurrencyCode>aaa</nms:UCCurrencyCode>

<nms:BarCodeId>aaaaaaaaaaaaaa</nms:BarCodeId>

<nms:BarCodeType>GTIN-13</nms:BarCodeType>

</nms:Item>

</nms:Items>

</nms:PODetails>

</nms:PurchaseOrder>

If you inspect the xml message, you can see that the ID field is missing. We will now run the XSLT processor against this xml instance and the Validating XSLT

​Below is the command

msxsl -o PO-Result.xml PO-Test.xml PO-Validating.xsl

And below is the contents of PO-Result.xml


<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?>

<schematron-output title=”” schemaVersion=”” phase=”#ALL” xmlns:sch=”http://www.ascc.net/xml/schematron&#8221; xmlns:nms=”http://www.nmstech.uk/ecommerce/schemas/v01″&gt;

<ns uri=” http://www.nmstech.uk/ecommerce/schemas/v01 ” prefix=”nms” />

<active-pattern name=”Check Existence”/>

<fired-rule id=”” context=”nms:PurchaseOrder” role=”” />

<failed-assert id=”” test=”nms:Id” role=”” location=”/nms:PurchaseOrder[1]”>

<text>PurchaseOrder Id is missing</text>

</failed-assert>

</schematron-output>

What we have done and achieved so far

1. Defined a schematron rules
2. Defined a Schematron Output schema
3. Executed the XSLT processor and got the results.

If you want to have schematron functionality for another flow, all you have to do is create the step 1, i.e. define schematron rules and that’s it, the rest all are the same.

Advertisements

Schematron Validation using XSLT in BizTalk – Part 1

This blog post is a 3 part series.

The source code for this blog post has been uploaded in the below location

http://code.msdn.microsoft.com/Schematron-validation-b4d892ee

BizTalk has an out of the box XML validation pipeline component to validate the semantics of the XSD schema and return the error message. However it has a lot of drawbacks
– can’t have custom error messages for failures
– can’t have all the error messages at one shot
Enter Schematron. Schematron is a language for making assertions about patterns found in XML documents. It is a rule based validation and is an ISO/IEC Standard.
There are currently many ways for implementing schematron. If we talk about BizTalk and .Net, there are two ways.
1. XPATH based implementation
2. XSLT based implementation

XPATH based in .Net Framework

Daniel Cazzulino, Microsoft XML MVP, designed a class library called Schematron.Net which provides classes for validating XML documents against the schematron schemas. This is done by the following
1. Embed Schematron rule patterns in BizTalk XSD schemas.
2. Write a custom pipeline component to call the Schematron.Net assembly to do the validation or use the Schematron Pipeline component available in Codeplex.

The above approach has some issues
1. There is a dependency to this class library. Although this library is open source, when you build BizTalk applications for enterprises, management doesn’t really like the idea of having 3rd Software libraries in the code. In case there are issues with the library, there will be no one to support.
2. A lot of XSLT functions are not available in this implementation

XSLT based Implementation

XSLT based Implementation works in two steps (Actually its 4 steps, however the extra 2 steps are actually required in complex scenarios)
1. The Schematron schema (.sch) is first turned into a validating XSLT stylesheet by transforming it with an XSLT stylesheet provided by Academica Sinica Computing Centre. These stylesheets (schematron-basic.xsl, schematron-message.xsl, schematron-report.xsl and conformance1-5.xsl) can be found at the Schematron site and the different stylesheets generate different output.
2. This validating stylesheet is then used on the XML instance document and the result will be a report that is based on the rules and assertions in the original Schematron schema.

xslt

Sending Soap with Attachments (SwA) using BizTalk – Part 1

This is a continuation to my previous article. Please have a look if you hadn’t seen it.

Sending Soap with Attachments (SwA) using BizTalk

Below are the steps that we have to follow for this exercise.

1. Create a custom message encoder which can send XML as an attachment in the SOAP envelope message.

2. Create a WCF service which can send the SOAP message with attachment.

3. Create the BizTalk application which can consume the custom WCF service.

Steps are easy isn’t. In this post we will only cover Step 1.

As I have told in the previous post, this custom encoder has been developed by Austrin interoperablity council and they have published the code in Codeplex. I have taken this code and removed some hard coded stuff and placed it in configurable properties.

I have uploaded the full source code below.

http://code.msdn.microsoft.com/Send-Soap-with-Attachments-3784e5f6

Shankar

Sending Soap with Attachments (SwA) using BizTalk

Soap With attachments is a very old technique and is still used in lot of legacy systems. More new technologies has emerged like MTOM which is widely used within the WCF arena. That said, there is no out of the box support within WCF to send a SOAP message with attachments.

An Austrian Microsoft Interoperability Council had a challenge to consume a webservice which expects to send a SOAP message with attachments.

http://blogs.msdn.com/b/mszcool/archive/2009/10/19/windows-communication-foundation-and-soap-with-attachments-message-encoder-built-in-interop-lab-with-svc-sozialversicherungs-chipkarten-betriebs-und-errichtungsgesellschaft-m-b-h.aspx

They have developed a custom encoder which can add the zip message as an attachment in the outgoinh SOAP envelope.

However I had a different challenge in consuming a webservice which requires me to send a soap payload which has an XML attachment. So I had to modify the custom encoder in such a way that I can attach a XML document. I had lot more challenges. The webservice which I had to consume was written in Java and the field appears as xsd:hexBinary in the WSDL file. So there is no way I can add a service reference / web reference to my project. I did tried that, but when I tried to send the message, the server was rejecting the message. The reason for this is Visual studio when generating the client code for the service, puts the type as xsd:base64binary.

So, I had to code the datacontracts, messagecontracts and servicecontracts based on the WSDL. Then I plugged in the code to use the encoder to attach the XML message. There is no direct way to do this from BizTalk. I wrote this entire logic in a WCF Service which acts as a client to the Target web service and acts as a Server for BizTalk. BizTalk application will consume this WCF service and send the SOAP message with attachment.

I am going to present an article with walkthrough in the next post.

Shankar

My contribution(BizTalk custom pipeline components) in technet website

I am happy to see that my contribution – BizTalk custom pipeline component has made it to list.

http://social.technet.microsoft.com/wiki/contents/articles/11679.biztalk-list-of-custom-pipeline-components-en-us.aspx

My component is listed in the Transformation section

Mapper Pipeline Component

This gave me an idea of developing the tranform concept in a pipeline component, wherein you don’t have to actually create a map for transforming the message.

http://www.codeproject.com/Articles/19228/Mapper-Pipeline-Component

Shankar

Build and Deploy Manager using BTDF

Fancy a build and deploy manager for BizTalk which uses BTDF in the background?

We use BTDF for building the BizTalk projects and most people now would have used TFS to do continuous integration which can utilize BTDF do build and deploy the projects.

However this is perfectly fine for automating the continuous integration process. When BizTalk projects has to be moved from development to test and production environments, manual installation steps are followed by deployment teams. For companies having a large number of projects and having the headache of following with the deployment team for the list of projects and release notes is inevitable. It sometimes require people from the dev team to sit with the deployment team to deploy the BizTalk applications.

I have developed a tool to assist with this issue. I have just started off with that tool. I will be constantly updating this tool to make it more generic.

Tool Background
Ideally this tool can be used by a build and release manager who can prepare the BizTalk MSI and deploy the applications in the respective environment without any BizTalk or BTDF knowledge

I have uploaded the project in codeplex. Its open source. So if you have any thoughts, pls share … shankar.sekar@gmail.com

http://buildanddeploybtdf.codeplex.com/

Thanks
Shankar