Schematron Validation using XSLT in BizTalk – Part 1

This blog post is a 3 part series.

The source code for this blog post has been uploaded in the below location

http://code.msdn.microsoft.com/Schematron-validation-b4d892ee

BizTalk has an out of the box XML validation pipeline component to validate the semantics of the XSD schema and return the error message. However it has a lot of drawbacks
– can’t have custom error messages for failures
– can’t have all the error messages at one shot
Enter Schematron. Schematron is a language for making assertions about patterns found in XML documents. It is a rule based validation and is an ISO/IEC Standard.
There are currently many ways for implementing schematron. If we talk about BizTalk and .Net, there are two ways.
1. XPATH based implementation
2. XSLT based implementation

XPATH based in .Net Framework

Daniel Cazzulino, Microsoft XML MVP, designed a class library called Schematron.Net which provides classes for validating XML documents against the schematron schemas. This is done by the following
1. Embed Schematron rule patterns in BizTalk XSD schemas.
2. Write a custom pipeline component to call the Schematron.Net assembly to do the validation or use the Schematron Pipeline component available in Codeplex.

The above approach has some issues
1. There is a dependency to this class library. Although this library is open source, when you build BizTalk applications for enterprises, management doesn’t really like the idea of having 3rd Software libraries in the code. In case there are issues with the library, there will be no one to support.
2. A lot of XSLT functions are not available in this implementation

XSLT based Implementation

XSLT based Implementation works in two steps (Actually its 4 steps, however the extra 2 steps are actually required in complex scenarios)
1. The Schematron schema (.sch) is first turned into a validating XSLT stylesheet by transforming it with an XSLT stylesheet provided by Academica Sinica Computing Centre. These stylesheets (schematron-basic.xsl, schematron-message.xsl, schematron-report.xsl and conformance1-5.xsl) can be found at the Schematron site and the different stylesheets generate different output.
2. This validating stylesheet is then used on the XML instance document and the result will be a report that is based on the rules and assertions in the original Schematron schema.

xslt

My contribution(BizTalk custom pipeline components) in technet website

I am happy to see that my contribution – BizTalk custom pipeline component has made it to list.

http://social.technet.microsoft.com/wiki/contents/articles/11679.biztalk-list-of-custom-pipeline-components-en-us.aspx

My component is listed in the Transformation section

Mapper Pipeline Component

This gave me an idea of developing the tranform concept in a pipeline component, wherein you don’t have to actually create a map for transforming the message.

http://www.codeproject.com/Articles/19228/Mapper-Pipeline-Component

Shankar

Build and Deploy Manager using BTDF

Fancy a build and deploy manager for BizTalk which uses BTDF in the background?

We use BTDF for building the BizTalk projects and most people now would have used TFS to do continuous integration which can utilize BTDF do build and deploy the projects.

However this is perfectly fine for automating the continuous integration process. When BizTalk projects has to be moved from development to test and production environments, manual installation steps are followed by deployment teams. For companies having a large number of projects and having the headache of following with the deployment team for the list of projects and release notes is inevitable. It sometimes require people from the dev team to sit with the deployment team to deploy the BizTalk applications.

I have developed a tool to assist with this issue. I have just started off with that tool. I will be constantly updating this tool to make it more generic.

Tool Background
Ideally this tool can be used by a build and release manager who can prepare the BizTalk MSI and deploy the applications in the respective environment without any BizTalk or BTDF knowledge

I have uploaded the project in codeplex. Its open source. So if you have any thoughts, pls share … shankar.sekar@gmail.com

http://buildanddeploybtdf.codeplex.com/

Thanks
Shankar

Replaying failed messages from BizTalk ESB portal Part 3

This post is in continuation with the series “Replaying failed messages from BizTalk ESB Portal”

If you haven’t seen the first post of this series, please use the below link to view the post. Part 2

Until now we have seen how to use BizTalk orchestration/receive port/send port to route failed messages to the ESB. But now we will how to replay the message from within the ESB Management Console. The ESB Portal is a ASP.Net application and the whole source code is made available to us by Microsoft.

The default replaying mechanism requires us to create either HTTP or WCF-HTTP adapters for replaying. I would like to go for a FILE adapter, as I would get an oppurtunity to view the message once it is replayed. To add support for a FILE adapter, we need to customize the ESB.Portal ASP.Net application. When you would have downloaded the ESB toolkit, you would have also got the source code for the ESB Portal and the corresponding web services applications.

Once you have loaded the solution, follow the below steps.

  1. In the ESB.Portal project, expand Faults and click MessageViewer.ascx file. Right click to view code or press F7.
  2. Navigate to the method PopulateReceiveLocationList().
  3. foreach (BizTalkOperationsService.BTReceiveLocation rcvLoc in sysStatus.ReceiveLocations)
                {
                    if (rcvLoc.Handler.ProtocolName.ToUpper().Equals("HTTP"))
                    {
                        rcvLoc.Address = "http://" + rcvLoc.Handler.Host.HostInstances[0].ServerName + rcvLoc.Address;
                        httpRcvLocs.Add(rcvLoc);
                        i++;
                    }
               }
    

    Replace the above code with

    foreach (BizTalkOperationsService.BTReceiveLocation rcvLoc in sysStatus.ReceiveLocations)
                {
                    if (rcvLoc.Handler.ProtocolName.ToUpper().Equals("HTTP"))
                    {
                        rcvLoc.Address = "http://" + rcvLoc.Handler.Host.HostInstances[0].ServerName + rcvLoc.Address;
                        httpRcvLocs.Add(rcvLoc);
                        i++;
                    }
                    else if (rcvLoc.Handler.ProtocolName.ToUpper().Equals("FILE"))
                    {
                        httpRcvLocs.Add(rcvLoc);
                        i++;
                    }
                }
    
  4. The above lines of code will include the FILE receive locations to appear in the ESB Management console.
    Now we need to navigate to the ResubmitMessage() method and add the below code

    else if (resubmitUrl.Contains("*.xml"))
                {
                    string url = resubmitUrl.Replace("\\", @"\");
                    url = url.Substring(0, url.IndexOf('*'));
    
                    System.Xml.XmlDocument doc = new System.Xml.XmlDocument();
                    StringBuilder envelopeMessage = new StringBuilder();
                    envelopeMessage.Append(((TextBox)this.MessageView.FindControl("messageBodyBox")).Text);
                    doc.LoadXml(envelopeMessage.ToString());
    
                    doc.Save(url + @"\" + System.Guid.NewGuid().ToString() + ".xml");
    
                    responseCode = "202";
                    responseMessage = "Successfully sumbitted to FILE Adapter";
    
                    result = true;
    
                }
    

    The above lines of code creates a xml document and drops it into the receive location URI that you have chosen. You will understand it better when you see everything in action.

  5. compile and project. If the project location is the same as the IIS directory location for this application, then you dont need to copy anything. If you are developing in a differnt location, then you can run the setup file that is also there in the solution to get your changes into the web application.
  6. We are all done and we can ahead and do our testing. Click the fault message in the Fault viewer and go to the message details. In the message viewer, click the edit link and the Text View control will now be enabled for you to edit the message. Go ahead and add the value 1 in the Count field and click Resubmit link.
  7. You will see a confirmation status in the ReSubmission status field. Go to Biztalk administration console and you could see that our orchestration would have completed without raising any exception. If you can enable tracking on the orchestration, you could now see the altered message.
  8. Have a look at the below picture for resubmitting a message

We have now completed replaying failed message from the ESB Management console without any context properties. If you observe the context message of the new one that you received, they would have lost its original context properties. In some circumnstance, you might need to replay a message with its original context property.

We will cover that in the next post.

Shankar

Replaying failed messages from BizTalk ESB portal Part 2

This post is in continuation with the series “Replaying failed messages from BizTalk ESB Portal”

If you haven’t seen the first post of this series, please use the below link to view the post. Part 1

In this Post, we will see a sample application. This solution will comprise a schema and an orchestration. We will see how we can route messages to the ESB from an orchestration. After doing this excercise we will also see how we can route failed messages from receive port and send port to the ESB.

So, lets first see how we can route messages from Orchestration.

  1. I created a sample application to demonstrate this. I have a xml schema which will be my source schema. The schema structure looks like this.


    See the Count element promoted in the schema.

  2. I added the below dll’s as reference to the project.
    C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Bin\Microsoft.Practices.ESB.ExceptionHandling.dll
    C:\Program Files (x86)\Microsoft BizTalk ESB Toolkit 2.1\Bin\Microsoft.Practices.ESB.ExceptionHandling.Schemas.Faults.dll
  3. Added an orchestration which will process the xml message. It receives the xml message using a receive shape and tries to assign the promoted value(Count) into a variable.
  4. As I explained in my previous post, I have added a scope that will have all my shapes. Currently there is only one shape in our business process. 7.Create a message in orchestration view and name it msgFaultMessage. Select the type as
    Microsoft.Practices.ESB.ExceptionHandling.Schemas.Faults.FaultMessage
  5. Add a construct shape and select the message that we have just added. Add a message assignment shape and the below code.
     msgFaultMessage = Microsoft.Practices.ESB.ExceptionHandling.ExceptionMgmt.CreateFaultMessage();
     msgFaultMessage.FailureCategory = “Process Questions”;
     msgFaultMessage.FaultCode = “”;
     msgFaultMessage.FaultDescription = ex.Message;
     msgFaultMessage.FaultSeverity = Microsoft.Practices.ESB.ExceptionHandling.FaultSeverity.Critical;
    Microsoft.Practices.ESB.ExceptionHandling.ExceptionMgmt.AddMessage(msgFaultMessage,msgQuestion);
    
  6. Add a send shape to send the fault message. Add a Logical send port in the orchestration and configure the binding as Direct with “Routing between ports…” option selected.
  7. Build the project and deploy the orchestration. Now to see the fault message routed to ESB, we need to create a source file for our orchestration to receive and process. Go ahead and generate an instance of the xml message. Modify the message and remove the value for the Count field. This will make the orchestration engine throw an error. That error which will caught by our exception handler and a fault message will be created. It will then be sent to Message box using direct send port.
  8. This is what you should in the BizTalk administration console.

  9. See how the message is showing up in ESB portal.

  10. You can click on the fault message to show its fault details in the fault viewer.

  11. You can find a grid at the bottom and this is the actual failed message. You can see a link for its messageid and if you click that it will give you a chance to see it message data and its context properties.

    Below is the message data

    Below is the context data

In this post I have explained how to route failed messages to ESB and see the fault messages within ESB Management console.

Messages can also be routed from receive ports and send ports. There is a check box in the properties and we can enable that to route fault messages to the message box.

In the next post, I will show you how to replay messages from the BizTalk ESB console.

Shankar

Could not enlist Send Port ‘XXXXX’. Exception from HRESULT: 0xC00CE557

This happens in a typical scenario, where you have a Send port which has a filter on it(Messaging based interface). The binding file that you used had this filter section, but unfortunately it had CRLF characters in
the filter section.
There are lot of posts in the internet which explains the fact that the filter part of the binding file has to be modified. If you would have used Visual Studio to edit the binding files, then it would have alter the structure of the filter section.

I will suggest to make the entire filter section in one line.

However the mistake that I did is, even after correcting the problem and re-importing the binding file, I still saw the problem. This is because, BizTalk doesnt do a clean up of the bindings, if you import a binding file on top of it. So you need to Un-Deploy the interface, Deploy the interface again and import the binding file.

Shankar

New Learnings from this present contract

I have had a really good time with this current contract and I have learned a lot new tools around BizTalk. It was one man army project and I had the privilege to try all new flavours in the BizTalk development. Below are the list of new stuff that I have learned in this project.

1. TFS 2010 Branching and Merging.

I have previously worked with TFS, but this time, I worked in a profressional way. I created projects suites based on standard development practices, with a MAINLINE branch and a DEVELOPMENT branch. The development branch was actually branched out from the MAINLINE. So now when all the coding is complete and when it is ready to be moved to UAT, I would merge my code to the MAINLINE branch. The Operations team would use the deployment files from the MAIN branch to deploy the code in UAT and PROD environments.

I also learned how to label the releases.  Although I new it before, I didnt know the significance of it. Now I know its full power. Awesome feature.

2. Deployment Framework for BizTalk

I have been following the legacy of using the Visual Studio and BizTalk adminstration console to deploy projects. I didnt know the power of MSBUILD until I went through a blog that discussed about the Deployment Framework for BizTalk. This piece of software is a open source project avaialble on CodePlex which will make your life easier when it comes to deployment. The software adds itself to the Visual Studio IDE making it easier for you to deploy your entire interface in a single click. You can also make use of MSBUILD scripts to deploy all the interfaces in a single go.

Consider that you are building a complex project and you have a common schemas project which is referenced by more than 10 interfaces. You have got a change request to change one schema, this means, you need to undeploy all the referenced applications, re-deploy the schema project, deploy the referenced applications and then import the binding files. This is a very error prone activity and you would often miss something. But with the MSBUILD scripts, you can just literally do this entire deployment with a single script file. Just run that and relax with a coffee or beer if your company permits.

3. BizUnit with Visual Studio Test Case Framework.

I had this long plan of writing BizUnit test classes and use the NUnit console to test the cases. I did got the oppurtunity in this project to use BizUnit. I downloaded the 4.0 version from CodePlex and re-built the source code. I then started looked for samples in google and to my bad luck all the samples that I looked were the ones which are deprecated in the 4.0 version. Luckily there were some samples out there and there was something included with the SDK itself. I created my own class files(without knowing the potential of the Test Case Framework in Visual Studio IDE) and starting executing the test cases. I was glad that it worked according to my command. But I still wanted to view the test results in a graphical format, that is because I have seen Nunit Console with all those green lights and red lights. But then I stumbled across a blog which taught me that I can use the Visual Studio IDE for creating test cases and use their framework to visualize the Test Cases. How Sweet is that. I migrated my code to the framework and starting testing my cases. I can now view the whole history of the Case details.

4. ESB exception console

When something failed in BizTalk, I always get the question, what happens next? Can you replay that message. I  used to hear this question a lot from a lot of people and will say the same answer that “once the message is consumed within BizTalk it is immutable and I can’t do anything further, you need to resend the message to us”. However after learning about the ESB exception management stuff, now I can tell anyone that we can replay the message and even alter the message before sending it. The microsoft team has delivered a lot of stuff which will enable people to replay the messages, however I would still need to customize the ESB portal to replay the messages via FILE adapter. Also the major problem is, you can’t replay a message which has context properties. If you do that, the context properties will be lost. I resolved that issue by adding a custom pipeline component and some code modification in ESB portal.

5. CAT Instrumentation tool

When you work with BizTalk you often realize that somewhere deep within the orchestration something has happened. To troubleshoot the issue you sometimes need more time than you have taken to develop that orchestration 😉 Thats where CAT instrumentation tool comes in picture. Again its a open source project available in code plex, it is a C# class library project. You just need to add reference to this dll and start instrumenting your statements within your expression shape. You also have a nice Instrumentation controller available in CodePlex which can control the Start and Stop of event tracing. The best thing is you can switch off/on debugging. This is a nice feature which we might need in Non-development environments, where we are in a hectic situation to analyze what is going on with that orchestration.

Hope I wont forget any of these nice tools in my next contract.

Cheers

Shankar