Preserving Input Sequence Order when Aggregating Debatched Messages

In a recent project, Jérémie wanted to improved one BizTalk solution where messages are debatched upon reception, processed independently and aggregated again before transmission to downstream processes or application partners.

This is sort of a classic Scatter-Gather pattern. The catch, however, is that each message, once debatched, comes with its own sequence number that must be maintained throughout the whole processing chain.

Since each stage in the process is independent of one another, we wanted to find the most flexible solution that still works in a decoupled fashion between the receive, the processing and the transmission sides.

The solution we came up with was to create a interim C# dictionary-like class to hold the messages as they are received and rely on the builtin enumerator to retrieve the messages in the correct sequence order.

A Simple Message Sorted Collection Class

The code below is a quick and simple collection class, designed to hold individual messages. This collection is keyed by an integer value that represents the order of the message in the original sequence of received messages.

[Serializable]
public class MessageSortedCollection : IEnumerable<XLANGMessage>
{
    private SortedDictionary<Int32, XLANGMessage> dictionary_ = new SortedDictionary<Int32, XLANGMessage>();

    public void Add(Int32 sequenceNumber, XLANGMessage message)
    {
        dictionary_.Add(sequenceNumber, message);
    }

    #region IEnumerable<XLANGMessage> Implementation

    public IEnumerator<XLANGMessage> GetEnumerator()
    {
        return dictionary_.Values.Select(di => di).GetEnumerator();
    }

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return (this as IEnumerable<XLANGMessage>).GetEnumerator();
    }

    #endregion
}

Storing Messages in Order

Using this helper class is very easy from your orchestration. In this post, I will demonstrate a simple scenario where each individual messages are first received by an upstream Orchestration for processing. Then, the following Orchestration, shown hereafter, is used for aggregation.

Let’s walk through this very simple Orchestration.

First, each debatched message must be stored temporarily, before it can be aggregated to the final message in the correct order.

For this purpose, we typically use a Singleton Orchestration whose purpose would be to perform some kind of processing and store them before the final aggregation.

In the preceding illustration, the Add Item Expression Shape holds code like this:

Collection.Add(InboundMessage.Body.sequence, InboundMessage); 

Where Collection is an Orchestration variable typed MessageSortedCollection and InboundMessage is an instance of a debatched message were the order in the original sequence is identified by a distinguished property.

Second, once all messages have been received – identified in the scenario by a timeout condition – its time to perform further processing before the final aggregation.

The following Orchestration snippet shows how one relies on the builtin enumeration behavior to iterate over the sequence of debatched messages in the correct order and submit them to a Send Pipeline.

The code in the Get Enumerator and Add to Pipeline Expression Shapes is straightforward:

Enumerator = Collection.GetEnumerator();

SendPipeline.Add(
    (Microsoft.XLANGs.BaseTypes.XLANGMessage) Enumerator.Current
);

Please, note that, since an IEnumerator object is not serializable, the iteration must happen within an atomic scope.

Next, the Send Pipeline is executed to perform the final aggregation and return the resulting message.

OutputMessage = new System.Xml.XmlDocument();

Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteSendPipeline(
      typeof(Microsoft.BizTalk.DefaultPipelines.XMLTransmit)
    , SendPipeline
    , OutputMessage);

This is a simple solution to a simple problem, but one that we BizTalker do not necessarily think about straight away. I hope this helps you deal with the similar situation in your projects.

About these ads