Pour la deuxième année consécutive, j’ai le plaisir d’animer une session sur la dernière version de SQL Server 2012 Master Data Services à l’occasion du salon Microsoft TechDays 2012, qui se tient au Palais des Congrès à Paris.

Cette année, je me suis vu confier l’animation de la session par Damien Cudel. Je me suis associé à Laurent Banon et mon collègue Radoine pour porter cette session sur un produit en lequel nous croyons vraiment en 2012 !

Le sujet, ambitieux, de la session est le suivant:

Avec SQL Server 2012 Master Data Services, la plateforme Application Server de Microsoft se dote d’un outil puissant de gestion de la donnée de référence. Nous allons montrer en quoi et comment cette solution, qui dispose d’une fonctionnalité de maintien de la qualité de la donnée, complète l’écosystème applicatif et permet de mettre en oeuvre des processus d’intégration, d’enrichissement et de diffusion des données de référence, en temps réel, entre les différentes partenaires applicatifs du système d’information, à l’intérieur comme à l’extérieur de l’entreprise.

Fortunately, Randal has been working towards producing a new 1.2.0.4 release of our PowerShell provider for BizTalk Server.

This version fixes a couple bugs and I invite everyone to test it out.

Thanks, Randal!

A big thanks to all my family, my friends and colleagues that supported me throughout the year.

I’m creating a custom Build system, in order to convert complete BizTalk solutions from source code, to Windows Installer .msi scripts. This involves not only building the projects (compiling), but also putting relevant managed assemblies to the Global Assembly Cache, deploying assemblies and third-party references to a BizTalk Application, applying bindings and exporting the resulting BizTalk application to a .msi file… I choose not to use the well-known and excellent BizTalk Deployment Framework, because, I’m primarily interested in a build system, tailored to a local development box in order to boost productivity.

In our shop, deployment is handled with a custom PowerShell-based solution that makes extensive use of the PowerShell provider for BizTalk. But for local development, there definitely was a need for improving productivity. For instance, by enabling incremental build support for BizTalk Server 2010 .btproj project files, we were able to dramatically cut the build times.

One of the things that broke when using our custom build system was to be able to perform the opposite operations – that is, undeploy – the BizTalk Application. This involves figuring out the correct chain of dependencies betweeen assemblies and remove then in the appropriate order from BizTalk.

Fortunately, MSBuild is extensible and it is not really difficult to include such features to your build scripts.

In order to alter you build system to include dependency-checking and correct build (and reverse cleanup) order, you need to apply the solution in two stages:

• First, you need to gather project dependencies
• Second, you need to figure out the correct build order

Gathering Project Dependencies

Figuring project dependencies relies on the use of ‘Project References’ in order to express project dependencies. This is what uses Visual Studio when building the projects from the solution. In order to gather the depencencies betweeen several projects, Visual Studio uses the ResolveProjectReferences builtin MSBuild task.

<Target Name="_ComputeProjectReference" Inputs="%(Project.Identity)" Outputs="%(Project.Identity)__force_output__">
  <MSBuild Projects="@(Project)" Targets="ResolveProjectReferences">
    <Output TaskParameter="TargetOutputs" ItemName="ResolvedProjectReferences" />
  </MSBuild>

  <ItemGroup>
    <ProjectReferenceItemSpec Include="%(ResolvedProjectReferences.OriginalProjectReferenceItemSpec)" />
  </ItemGroup>

 <ItemGroup>
    <ProjectReference Include="@(Project)">
      <ProjectReference>@(ProjectReferenceItemSpec)</ProjectReference>
    </ProjectReference> </ItemGroup>
</Target>

Hint: scanning project dependencies actually triggers a build and takes time. This can sometimes be undesirable. In order to speed things up a litte and avoid redundant builds, you can persist the project dependencies in a temporary file, with the WriteLinesToFile MSBuild task, and read the cached information on subsequent builds with the ReadLinesFromFile task.

Figuring out the Correct Build Order

When all project references have been gathered, figuring out the correct build order is trivial. In fact, this subject has been beaten to death already, and is an application of Graph Theory. More specifically, project dependencies form a Directed Acyclic Graph. The action of ordering the projects for successful build is called a Topological Sort.

The following simple custom class includes a direct translation of the corresponding Wikipedia entry on the subject:

using System;
using System.Collections.Generic;

namespace System.Collections.Generic
{
    public class Graph<T>
    {
        #region Constructors

        private IList<T> nodes_ = new List<T>();
        private IDictionary<Int32, List<Int32>> edges_ = new Dictionary<Int32, List<Int32>>();

        #endregion

        #region Operations

        public void AddEdge(T item, T dependency)
        {
            int index_item = AddNode(item);
            int index_dependency = AddNode(dependency);

            edges_[index_item].Add(index_dependency);
        }

        public Int32 AddNode(T item)
        {
            if (!nodes_.Contains(item))
                nodes_.Add(item);

            int index = IndexOf(item);

            if (!edges_.ContainsKey(index))
                edges_.Add(index, new List<Int32>());

            return index;
        }

        public T[] Sort()
        {
            IList<T> Nodes = new List<T>(nodes_);
            IDictionary<Int32, List<Int32>> Edges = new Dictionary<Int32, List<Int32>>(edges_);

            // L ← Empty list that will contain the sorted elements
            // S ← Set of all nodes with no incoming edges

            List<T> L = new List<T>(Nodes.Count);
            List<Int32> S = new List<int>(Nodes.Count);
            foreach (Int32 index in Edges.Keys)
                if (Edges[index].Count == 0)
                    S.Add(index);

            // while S is non-empty do
            // remove a node n from S
            // insert n into L
            // for each node m with an edge e from n to m do
            // remove edge e from the graph
            // if m has no other incoming edges then
            // insert m into S

            while (S.Count != 0)
            {
                T node = Nodes[S[0]];
                S.RemoveAt(0);

                L.Add(node);

                Int32 index_from = Nodes.IndexOf(node);

                foreach (KeyValuePair<Int32, List<Int32>> item in Edges)
                {
                    if (item.Value.Contains(index_from))
                    {
                        item.Value.Remove(index_from);
                        if (item.Value.Count == 0)
                            S.Add(item.Key);
                    }
                }
            }

            // if graph has edges then
            // output error message (graph has at least one cycle)
            // else 
            // output message (proposed topologically sorted order: L)

            foreach (List<Int32> item in Edges.Values)
                if (item.Count != 0)
                    throw new ApplicationException("Circular dependency detected!");

            return L.ToArray();
        }

        public T[] Sort(bool reverse)
        {
            T[] array = Sort();
            if (reverse)
                Array.Reverse(array);
            return array;
        }

        #endregion

        #region Implementation

        private Int32 IndexOf(T item)
        {
            return nodes_.IndexOf(item);
        }

        #endregion
    }
}

A Custom MSBuild Task for Topological Sort

The preceding class can be used in a custom MSBuild task in order to sort the corresponding projects in the order (or reverse order) of dependencies:

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;

using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

namespace BizTalkFactory.Automation
{
    public class SortDependentProject : Task
    {
        private bool most_dependent_ = false;
        private ITaskItem[] outputs_ = new ITaskItem[]{};

        [Required]
        public ITaskItem[] Project { get; set; }

        [Output]
        public ITaskItem[] TargetOutputs
        {
            get
            {
                return outputs_;
            }
            set
            {
                outputs_ = value;
            }
        }

        public Boolean MostDependentFirst
        {
            get { return most_dependent_; }
            set { most_dependent_ = value; }
        }

        #region Overrides

        public override bool Execute()
        {
            if (Project.Length == 0)
                outputs_ = Project;

            Graph<String> graph = new Graph<string>();

            foreach (ITaskItem item in Project)
            {
                string dependencies = (string)item.GetMetadata("ProjectReference");
                string[] references = Canonicalize(Path.GetDirectoryName(item.ItemSpec), dependencies);
                foreach (string reference in references)
                    if (String.IsNullOrEmpty(reference))
                        graph.AddNode(item.ItemSpec);
                    else
                        graph.AddEdge(item.ItemSpec, reference);
            }

            List<ITaskItem> taskItems = new List<ITaskItem>(Project.Length);

            foreach (String item in graph.Sort(most_dependent_))
            {
                TaskItem taskItem = new TaskItem();
                taskItem.ItemSpec = item;

                ITaskItem sourceItem = Project.Single(element => element.ItemSpec == taskItem.ItemSpec);
                sourceItem.CopyMetadataTo(taskItem);

                taskItems.Add(taskItem);
            }

            outputs_ = taskItems.ToArray();
            return true;
        }

        #endregion

        #region Implementation

        private string[] Canonicalize(string relativeRoot, string path)
        {
            if (String.IsNullOrEmpty(path))
                return new string[]{ path };

            List<String> strings = new List<String>();

            foreach (string item in path.Split(new char[] { ';' }))
            {

                if (Path.IsPathRooted(item) && File.Exists(item))
                    strings.Add(item);

                string resolved = Path.GetFullPath(Path.Combine(relativeRoot, item));

                if (File.Exists(resolved))
                    strings.Add(resolved);
            }

            return strings.ToArray();
        }

        #endregion
    }
}

This custom task would be used simply in an MSBuild project file like so:

    <!-- Calculate projects in order of dependencies -->

    <MSBuild Projects="$(MSBuildProjectFile)" Targets="_ComputeProjectReference" />

    <CustomTask.Automation.SortDependentProject
        Project="@(ProjectReference)"
        MostDependentFirst="$(Clean)">
      <Output TaskParameter="TargetOutputs" ItemName="ProjectDependencies" />
    </CustomTask.Automation.SortDependentProject>

    <ItemGroup>
      <Project Remove="@(Project)" />
      <Project Include="@(ProjectDependencies)" />
    </ItemGroup>

For some reason, BizTalk projects need to be compiled in two phases.

The first pass compiles schemas, maps and pipelines, and produces an intermediate managed assembly. The second pass deletes the intermediate assembly, compiles the orchestrations and produces the final output assembly.

This behavior is hardcoded in the $(MSBuildExtensionsPath)\Microsoft\BizTalk\BizTalkC.targets file.

Unfortunately, this breaks incremental compilation of large solutions that make use of BizTalk Server projects because it forces a recompile each time a project is being built. Fortunately, there is a solution.

MSBuild being extensible, it is possible to override the default (incorrect) targets and fix this. First, create an appropriate file, say BizTalkCustom.targets to host your customizations, in any location of your choice :

  <!-- Rerun the build process (second pass) if the project hosts orchestrations -->
  <!-- The extra << and @(CSharpOutputFromXLang)!='' >> condition prevents this task -->
  <!-- from taking place if the project does not contain any orchestrations.  -->
  <Target Name="SecondPass" Condition="$(SecondBuild)!=true and $(TempAssemblyOnly)!=true and @(XLang)!=''">
    <MSBuild Projects="$(MSBuildProjectFile)" Properties="SecondBuild=true" />
  </Target>

  <!-- Compile XLang/s orchestration -->
  <!-- Notice that is the MSBuild system has decided to run this Target, -->
  <!-- both intermediately compiled files and intermediately compiled assembly are deleted. -->
  <Target
    Name="CompileODX"
    Condition="$(SecondBuild)==true"
    Inputs="@(XLang);$(MSBuildAllProjects);$(ClrTypesAssembly)"
    Outputs="$(BuildDone)">

    <!-- Delete previously generated C# files from XLang compilation -->
    <Delete Files="@(IntermediateAssembly)" />
    <Delete Files="@(CSharpOutputFromXLang)" />

    <XLangTask XLangItems="@(XLang)"
               ProjectReferences="@(ReferencePath)"
               WarningLevel="$(WarningLevel)"
               BpelCompliance="$(BpelCompliance)"
               DefineConstants="$(DefineConstants)"
               TreatWarningsAsErrors="$(TreatWarningsAsErrors)"
               TempAssembly="$(ClrTypesAssembly)"
               OutputDirectory="$(XLangOutputPath)">

    </XLangTask>
  </Target>

The first part of the solution is to modify the <SecondPass> Target in order to add an extra test in the <Condition> attribute. This prevents the SecondPass from even taking place in case your .btproj does not host orchestrations.

The second part of the solution is to defer the removal of the intermediate managed assembly to the <CompileODX> target. Indeed, this target correctly handles incremental compilation, so it does not run if all files are already up-to-date.

In order to use the overriden targets in your compilations, you need to slightly modify your .btproj project files, and import the preceding .targets file, like so:

<Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />
<Import Project="$(MSBuildExtensionsPath)\Microsoft\BizTalk\BizTalkC.targets" />
<Import
    Condition=" Exists('$(CustomExtensionsPath)BizTalkCustom.targets') "
    Project="$(CustomExtensionsPath)BizTalkCustom.targets" />

Where $(CustomExtensionsPath) represents the path to your custom extensions project files.

This concludes a session-packed week that gave all of us, developers, architects, managers and the world at large what the ambitions of Microsoft are for the future.

It’s an interesting and exciting moment and situation to be in, because we are the ones with this opportunity to lead the transformation of the way computers and devices are used, connected and consumed.

But the critical main thought was delivered as part of the day’s opening Keynote:

I think this summarizes very well what the strategy of Microsoft comes down to for the years ahead.

Keynote

What we’ve seen today:

A showcase of the new Metro-Style user interface. This touch-first interface, first seen on the Zune and recent Windows Phone devices, has been significantly enhanced. It runs on many devices, in a variety of shapes and sizes, including ARM-based tablets and phones.

What powers the new Metro user-interface is the pervasive use of HTML 5 and Javascript, supported nativelely by the new Internet Explorer 10 rendering engine.

A whole host of improvements, both in the core architecture of Windows and in the user interface. Because, indeed, the existing user interface is still there and protects the massive investment that has been made on Windows over the years.

Last, but not least, Microsoft has given away the full suite of Developer’s Preview Software stack – Windows 8, Visual Studio 11, etc. – bundled on, what they call the Developer’s Preview Hardware: a brand new Core i5 Samsung Windows 8 Tablet!

This most anticipated piece of hardware was demoed live and will definitely prove to be a major contender against competing offerings from Apple and Google.

By giving away the Developer’s Preview Hardware tablet, Microsoft hopes to create a vast ecosytem of applications, ready to be downloaded from the upcoming Windows Store when the platform ships, probably around spring next year.

In fact, by bringing support for both Multi-Touch and Stylus-based Input to Windows, and allow for seamless co-existing Metro-Style and Desktop applications, this platform will certainly appeal to professionnals and consumers alike with a compelling set of applications.

More famously known for its DisneyLand Resort, Anaheim is getting busy today, with many attendees converging to register at the Convention Center, in preparation of the upcoming conference, where Microsoft promises to unveil the future of Windows.

In situations like these, it is customary to have Visual Studio generate an XSLT file from the BizTalk map, and use Visual Studio’s builtin support for debugging XSLT. This allows you to put breakpoints, inspect the content of variables and precisely monitor the transformation step by step.

Custom Extension Objects

There are two cases, however, that prevent you from debugging the underlying XSLT from within Visual Studio :

The first case involves the use of the various database functoids, for instance the cross-reference GetCommonValue, GetCommonID, GetApplicationValue and GetApplicationID functoids. These, are implemented in terms of external functions, in a separate assembly, that is itself referenced from the generated map.

The second case involves the use of the Scripting functoid to call an external function. This is a variation of the first case in which you specify the function you want to call.

In order to execute such maps, the BizTalk Mapper uses the XslTransform class by supplying a list of all the classes whose methods are required to carry out the XSLT transformation, along with their associated namespace prefixes.

The set of such custom classes used to provide external logic to XLST transformations is referred to as Custom XSLT Extension Objects. At design-time, these Custom Extension Objects are represented as an additional file that contains the associations between the declared XML namespace prefixes and the fully qualified .Net type names of the classes which contain the methods called from the map.

When you invoke the “Validate Map” context-menu option on a BizTalk Map in Visual Studio, you will notice that BizTalk generates this file, with a _extxml.xml suffix along with the associated XSLT.

Most of the time, this file is empty, for basic maps. This is not the case, however, if the map makes use of external functions in separate assemblies, as alluded to in the two cases described above.

Debugging XSLT with Custom Extension Objects

Unfortunately, such maps seemingly cannot be debugged inside Visual Studio without modifications. If you use Visual Studio’s builtin XSLT debugger and attempt to step into the underlying XSLT file, you will encounter an error such as this one:

---------------------
Cannot find the script or external object that implements prefix
'http://schemas.microsoft.com/BizTalk/2003/ScriptNS0'. "

Even though Visual Studio generates both an XSLT file and a _EXTXML.XML file, it seems there is no way to instruct Visual Studio about the custom extension objects file, which makes the debugging experience cumbersome and very frustrating.

Fortunately, there is a way!

Although there are no options in the Visual Studio GUI, all the support is there in the .Net Framework. All there is do is write a little C# program to do the trick.

A Simple Function to Debug Custom XLST Maps

The following function is very simple. In real life, this function would be compiled into a C# program with additional logic to accept and parse command-line arguments, in order to be able to supply paths to the stylesheet, the extension objects file and the input and output documents. This is no rocket science.

Here is the code:

using System;
using System.Collections;
using System.IO;
using System.Reflection;
using System.Text;
using System.Xml;
using System.Xml.Xsl;

class DebugHelper
{
  static void Main(string[] args)
  {
    string input = @"...\input.xml";
    string output = Path.ChangeExtension(input, ".out");

    string stylesheet = @"...\stylesheet.xslt";
    string extension_object = @"...\extension_extxml.xml";

    string result = TransformXslt(input, stylesheet, extension_object);
  }

  private static string TransformXslt(string document, string stylesheet, string extension)
  {
    return TransformXslt(document, stylesheet, ParseExtension(extension));
  }

  private static string TransformXslt(string document, string stylesheet, object[] extension)
  {
    XslCompiledTransform transform = new XslCompiledTransform(true);
    transform.Load(stylesheet, new XsltSettings(true, true), null);

    XsltArgumentList arguments = new XsltArgumentList();

    for (int index = 0; index < extension.Length; index += 2)
    {
      arguments.AddExtensionObject(
        extension[index] as string,
        extension[index + 1]
        );
    }

    StringBuilder output = new StringBuilder();

    using (XmlWriter writer = XmlWriter.Create(output, transform.OutputSettings))
      transform.Transform(document, arguments, writer);

    return output.ToString();
  }

  private static object[] ParseExtension(string extension)
  {
    if (String.IsNullOrEmpty(extension))
      return new object[]{};

    XmlDocument document = new XmlDocument();
    document.Load(extension);

    ArrayList extensions = new ArrayList();

    foreach (XmlNode node in document.SelectNodes("/ExtensionObjects/ExtensionObject"))
    {
      string extension_namespace = node.Attributes["Namespace"].Value;
      string extension_assembly = node.Attributes["AssemblyName"].Value;
      string extension_class = node.Attributes["ClassName"].Value;
      string assembly_qualified_name = String.Format("{0}, {1}"
        , extension_class
        , extension_assembly
        );

      object extension_object = Activator.CreateInstance(Type.GetType(assembly_qualified_name));

      extensions.Add(extension_namespace);
      extensions.Add(extension_object);
    }

    return extensions.ToArray();
  }
}

A couple of points of interest need clarification.

First, the most crucial step, is to use the XslCompiledTransform constructor with a first parameter that indicates whether to execute the map in a debugger. This is actually what does the trick in this program.

When you run the program, it will launch the debugger and break at the beginning of the map. You can add this project to your Visual Studio solution, and you will be able to step into the map directly from the same Visual Studio instance.

Neat.

Secondly, what distinguishes this program from Visual Studio’s builtin XSLT debugger is the ability to actually make use of Custom XSLT Extension Objects.

The ParseExtensions function has been designed to consume an XML file, whose path is specified in the first argument, that has the same format as the ones generated by Visual Studio when using the “Validate Map” option.