Generic FlatFile Receive and Transmit Pipelines

Enterprise Integration Platforms such as Microsoft BizTalk Server offer compelling solutions to rationalize the transmission of data between different heterogeneous third-party systems. When working with legacy systems particularly, it is not uncommon for the data to be exchanged via the use of batch files.

Working with such « flat files » requires the use of a disassemble stage during acquisition in order to transform the incoming message to a more amenable XML format. Similarly, during the transmission to external systems, those messages may need to be serialized back to a flat file format.

As you know, assembling or disassembling such flat files to XML messages requires the use of the « Flat File Schema Extension ». With the « Flat File Schema Wizard » one can design an XML schema that describes the data as it appears in the corresponding flat file. This schema is then specified in a Flat File Disassembler pipeline component in a receive pipeline or a Flat File Assembler pipeline component in a transmit pipeline.

From an application development standpoint, using flat file assembling and disassembling pipelines requires the maintainance and deployment of two BizTalk artifacts : one for the schema and the other for the pipeline itself.

During one of my earlier projects, I had to implement the communication with several third-party systems requiring the use of flat files. In order to keep the maintenance to a minimum, it quickly became clear that I had to reduce the number of deployed artifacts.

Creating Generic FlatFileReceive and FlatFileTransmit pipelines

One little known feature introduced with BizTalk Server 2004 was the possibility to modify the configuration of a pipeline even after it is deployed. The modified parameters are stored, together with the bindings of the corresponding BizTalk solution. However, at the time, there was no suitable user interface to access this feature.

With BizTalk Server 2006 and beyond, the possibility to modify this per-instance configuration data is integrated into the management console and is accessible in the general configuration property page of the receive locations and send ports by clicking the little « browse » button located at the right of the pipeline selection dropdown list.

All the flat file assembling and disassembling pipelines are virtually identical, with regards to the pipeline components that are used. Fortunately, in order to limit the number of such pipelines we need to maintain and deploy it is possible to create and configure a single generic version of such pipelines (or at most two, depending upon the direction of the data transmission).

However, it is actually impossible to create and deploy a flat file disassembling (or assembling) pipeline without specifying a specific schema in the "DocumentSpecName" property. To circumvent this problem, I recommand specifying the builtin ‘Any’ schema, that is available in the Microsoft.XLANGs.BaseTypes assembly. Note that a reference to this assembly must be added to the Visual Studio project.

Once deployed, the per-instance configuration of the pipeline must be customized. In the "Configure Pipeline" dialog box, specify the fully qualified name of the associated schema, followed by the fully qualified name of the .Net assembly where it is hosted.

The "DocumentSpecName" property looks like this:

<type name>, <assembly name>, <version>, <culture>, <signature>

For instance :


Pros and Cons

The key advantage of this solution lies on the substantial reduction of the number of BizTalk pipelines we need to maintain and deploy. The only thing to worry about is the bindings configuration after the solution is deployed.

However this can be seen as a disadvantage. Indeed, this configuration consists in modifying the schemas used to transform the messaged as they flow through the application. In essence this represents the very nature of the information that are received or sent.

It might be argued that this configuration must not be delegated to systems administrators during the deployment and must be specified during the design and development phases.

I personally think that, however simple they are, such generic pipelines are a very useful tool for limiting the number of artifacts that need to be deployed. Specifically, I make extensive use of this solutions in the unit testing phase, to simulate external systems with a simple combination of CBR solutions used as stubs.

I think that this solution if best suited for BizTalk solutions because it makes it more maintainable and evolutive in the end.

You mileage may vary.

This entry was posted in BizTalk. Bookmark the permalink.

One Response to Generic FlatFile Receive and Transmit Pipelines

  1. Michel says:

    Very useful tip

Comments are closed.