Building Multiple Projects in Order with MSBuild

When building projects grouped in a solution, Visual Studio figures out the project dependencies and respect the build order. However, if you’re not using Visual Studio .sln solution files when building a set of projects with MSBuild, the build order is left to the author of the MSBuild script.

I’m creating a custom Build system, in order to convert complete BizTalk solutions from source code, to Windows Installer .msi scripts. This involves not only building the projects (compiling), but also putting relevant managed assemblies to the Global Assembly Cache, deploying assemblies and third-party references to a BizTalk Application, applying bindings and exporting the resulting BizTalk application to a .msi file… I choose not to use the well-known and excellent BizTalk Deployment Framework, because, I’m primarily interested in a build system, tailored to a local development box in order to boost productivity.

In our shop, deployment is handled with a custom PowerShell-based solution that makes extensive use of the PowerShell provider for BizTalk. But for local development, there definitely was a need for improving productivity. For instance, by enabling incremental build support for BizTalk Server 2010 .btproj project files, we were able to dramatically cut the build times.

One of the things that broke when using our custom build system was to be able to perform the opposite operations – that is, undeploy – the BizTalk Application. This involves figuring out the correct chain of dependencies betweeen assemblies and remove then in the appropriate order from BizTalk.

Fortunately, MSBuild is extensible and it is not really difficult to include such features to your build scripts.

In order to alter you build system to include dependency-checking and correct build (and reverse cleanup) order, you need to apply the solution in two stages:

  • First, you need to gather project dependencies
  • Second, you need to figure out the correct build order

Gathering Project Dependencies

Figuring project dependencies relies on the use of ‘Project References’ in order to express project dependencies. This is what uses Visual Studio when building the projects from the solution. In order to gather the depencencies betweeen several projects, Visual Studio uses the ResolveProjectReferences builtin MSBuild task.

<Target Name="_ComputeProjectReference" Inputs="%(Project.Identity)" Outputs="%(Project.Identity)__force_output__">
  <MSBuild Projects="@(Project)" Targets="ResolveProjectReferences">
    <Output TaskParameter="TargetOutputs" ItemName="ResolvedProjectReferences" />
  </MSBuild>

  <ItemGroup>
    <ProjectReferenceItemSpec Include="%(ResolvedProjectReferences.OriginalProjectReferenceItemSpec)" />
  </ItemGroup>

 <ItemGroup>
    <ProjectReference Include="@(Project)">
      <ProjectReference>@(ProjectReferenceItemSpec)</ProjectReference>
    </ProjectReference> </ItemGroup>
</Target>

Hint: scanning project dependencies actually triggers a build and takes time. This can sometimes be undesirable. In order to speed things up a litte and avoid redundant builds, you can persist the project dependencies in a temporary file, with the WriteLinesToFile MSBuild task, and read the cached information on subsequent builds with the ReadLinesFromFile task.

Figuring out the Correct Build Order

When all project references have been gathered, figuring out the correct build order is trivial. In fact, this subject has been beaten to death already, and is an application of Graph Theory. More specifically, project dependencies form a Directed Acyclic Graph. The action of ordering the projects for successful build is called a Topological Sort.

The following simple custom class includes a direct translation of the corresponding Wikipedia entry on the subject:

using System;
using System.Collections.Generic;

namespace System.Collections.Generic
{
    public class Graph<T>
    {
        #region Constructors

        private IList<T> nodes_ = new List<T>();
        private IDictionary<Int32, List<Int32>> edges_ = new Dictionary<Int32, List<Int32>>();

        #endregion

        #region Operations

        public void AddEdge(T item, T dependency)
        {
            int index_item = AddNode(item);
            int index_dependency = AddNode(dependency);

            edges_[index_item].Add(index_dependency);
        }

        public Int32 AddNode(T item)
        {
            if (!nodes_.Contains(item))
                nodes_.Add(item);

            int index = IndexOf(item);

            if (!edges_.ContainsKey(index))
                edges_.Add(index, new List<Int32>());

            return index;
        }

        public T[] Sort()
        {
            IList<T> Nodes = new List<T>(nodes_);
            IDictionary<Int32, List<Int32>> Edges = new Dictionary<Int32, List<Int32>>(edges_);

            // L ← Empty list that will contain the sorted elements
            // S ← Set of all nodes with no incoming edges

            List<T> L = new List<T>(Nodes.Count);
            List<Int32> S = new List<int>(Nodes.Count);
            foreach (Int32 index in Edges.Keys)
                if (Edges[index].Count == 0)
                    S.Add(index);

            // while S is non-empty do
            // remove a node n from S
            // insert n into L
            // for each node m with an edge e from n to m do
            // remove edge e from the graph
            // if m has no other incoming edges then
            // insert m into S

            while (S.Count != 0)
            {
                T node = Nodes[S[0]];
                S.RemoveAt(0);

                L.Add(node);

                Int32 index_from = Nodes.IndexOf(node);

                foreach (KeyValuePair<Int32, List<Int32>> item in Edges)
                {
                    if (item.Value.Contains(index_from))
                    {
                        item.Value.Remove(index_from);
                        if (item.Value.Count == 0)
                            S.Add(item.Key);
                    }
                }
            }

            // if graph has edges then
            // output error message (graph has at least one cycle)
            // else 
            // output message (proposed topologically sorted order: L)

            foreach (List<Int32> item in Edges.Values)
                if (item.Count != 0)
                    throw new ApplicationException("Circular dependency detected!");

            return L.ToArray();
        }

        public T[] Sort(bool reverse)
        {
            T[] array = Sort();
            if (reverse)
                Array.Reverse(array);
            return array;
        }

        #endregion

        #region Implementation

        private Int32 IndexOf(T item)
        {
            return nodes_.IndexOf(item);
        }

        #endregion
    }
}

A Custom MSBuild Task for Topological Sort

The preceding class can be used in a custom MSBuild task in order to sort the corresponding projects in the order (or reverse order) of dependencies:

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;

using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

namespace BizTalkFactory.Automation
{
    public class SortDependentProject : Task
    {
        private bool most_dependent_ = false;
        private ITaskItem[] outputs_ = new ITaskItem[]{};

        [Required]
        public ITaskItem[] Project { get; set; }

        [Output]
        public ITaskItem[] TargetOutputs
        {
            get
            {
                return outputs_;
            }
            set
            {
                outputs_ = value;
            }
        }

        public Boolean MostDependentFirst
        {
            get { return most_dependent_; }
            set { most_dependent_ = value; }
        }

        #region Overrides

        public override bool Execute()
        {
            if (Project.Length == 0)
                outputs_ = Project;

            Graph<String> graph = new Graph<string>();

            foreach (ITaskItem item in Project)
            {
                string dependencies = (string)item.GetMetadata("ProjectReference");
                string[] references = Canonicalize(Path.GetDirectoryName(item.ItemSpec), dependencies);
                foreach (string reference in references)
                    if (String.IsNullOrEmpty(reference))
                        graph.AddNode(item.ItemSpec);
                    else
                        graph.AddEdge(item.ItemSpec, reference);
            }

            List<ITaskItem> taskItems = new List<ITaskItem>(Project.Length);

            foreach (String item in graph.Sort(most_dependent_))
            {
                TaskItem taskItem = new TaskItem();
                taskItem.ItemSpec = item;

                ITaskItem sourceItem = Project.Single(element => element.ItemSpec == taskItem.ItemSpec);
                sourceItem.CopyMetadataTo(taskItem);

                taskItems.Add(taskItem);
            }

            outputs_ = taskItems.ToArray();
            return true;
        }

        #endregion

        #region Implementation

        private string[] Canonicalize(string relativeRoot, string path)
        {
            if (String.IsNullOrEmpty(path))
                return new string[]{ path };

            List<String> strings = new List<String>();

            foreach (string item in path.Split(new char[] { ';' }))
            {

                if (Path.IsPathRooted(item) && File.Exists(item))
                    strings.Add(item);

                string resolved = Path.GetFullPath(Path.Combine(relativeRoot, item));

                if (File.Exists(resolved))
                    strings.Add(resolved);
            }

            return strings.ToArray();
        }

        #endregion
    }
}

This custom task would be used simply in an MSBuild project file like so:

    <!-- Calculate projects in order of dependencies -->

    <MSBuild Projects="$(MSBuildProjectFile)" Targets="_ComputeProjectReference" />

    <CustomTask.Automation.SortDependentProject
        Project="@(ProjectReference)"
        MostDependentFirst="$(Clean)">
      <Output TaskParameter="TargetOutputs" ItemName="ProjectDependencies" />
    </CustomTask.Automation.SortDependentProject>

    <ItemGroup>
      <Project Remove="@(Project)" />
      <Project Include="@(ProjectDependencies)" />
    </ItemGroup>
This entry was posted in BizTalk, MSBuild. Bookmark the permalink.

One Response to Building Multiple Projects in Order with MSBuild

  1. Pablo Trujillo says:

    This is a great job, I just copied and used on my msbuild process, with small changes it allow me to find a project reference which is not included in a solution file, by default DevEnv stop when a project reference does not exists in the solution but MSBuild simply compile it without any message, with this job I’m able to notify when a project reference does not exists in the solution

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s