Capturing REST API Calls with an OWIN Middleware

Update:I have set a GitHub repository hosting an updated and more complete version of this code.

With the advent of the Cloud First, Mobile First world we live in, you are aware that REST APIs are becoming more and more important every day. In one of our projects, we wanted to capture details of every access to our APIs for security and audit purposes. Ideally, we wanted to build some kind of interception mechanism, so that the developer of the API does not have to deal with the intricacies and the details of the capturing process.

This is a form of Aspect Oriented Programming (AOP), where the concern of capturing API calls is orthogonal to the concern of performing the work the API is designed for. There are many ways to achieve this, and many frameworks to complete this task.

Fortunately, we are exposing our REST APIs using Katana, which is Microsoft’s open-source implementation of the OWIN protocol. OWIN has the concept of Middleware, which are layered on top of one another, in order to perform some processing. One of these Middleware handles processing and routing requests to ASP.NET WebAPI Controllers:

namespace App
{
    using System;
    using System.Web.Http;
    using Microsoft.Owin.Hosting;
    using Owin;
    
    class Program
    {
        static void Main(string[] args)
        {
            const string address = "http://localhost:12345";

            using (WebApp.Start(address))
            {
                Console.WriteLine("Listening requests on {0}.", address);
                Console.WriteLine("Please, press ENTER to shutdown.");
                Console.ReadLine();
            }
        }
    }

    public sealed class Startup
    {
        private readonly HttpConfiguration configuration_ = new HttpConfiguration();

        public void Configuration(IAppBuilder application)
        {
            application.UseWebApi(configuration_);

            configuration_.MapHttpAttributeRoutes();
        }
    }
}

Creating an OWIN Middleware

In order to capture REST API calls, we will be creating a custom OWIN Middleware. Middleware in OWIN form a chain and are included in the processing pipeline. Each Middleware has a reference to and is responsible for calling the next Middleware in the chain as part of its processing.

The absolute minimal do-nothing OWIN Middleware looks like so:

namespace Middleware
{
    using Microsoft.Owin;

    public sealed class TrackingMiddleware : OwinMiddleware
    {
        public TrackingMiddleware(OwinMiddleware next)
            : base(next)
        {
        }

        public override async System.Threading.Tasks.Task Invoke(IOwinContext context)
        {
            await Next.Invoke(context);
        }
    }
}

Our Tracking Middleware is layered on top of the chain, so that it is called first when an HTTP request comes in and called last when the corresponding HTTP response comes back from the Web API. Thus, our Middleware has a chance to track details about both the request and the response.

    public sealed class Startup
    {
        ...
        public void Configuration(IAppBuilder application)
        {
            application.Use<Middleware.TrackingMiddleware>();
            application.UseWebApi(configuration_);

            configuration_.MapHttpAttributeRoutes();
        }
    }

But first, some book-keeping

In order to make this post more meaningful, we need to have some boilerplate code in place first. So, for illustration purposes, let’s create a very simple WebApi controller that will be used in the rest of this article.

namespace WebApi
{
    using System;
    using System.Threading.Tasks;
    using System.Web.Http;

    [RoutePrefix("api/custom")]
    public sealed class CustomController : ApiController
    {
        [HttpGet]
        [Route("resources")]
        public IHttpActionResult GetAllResources()
        {
            var collection = new[]
            {
                "one",
                "two",
                "three",
            };

            return Ok(collection);
        }

        [HttpPost]
        [Route("post")]
        public async Task<IHttpActionResult> PostStream()
        {
            // consume the stream, to simulate processing
            using (var stream = await Request.Content.ReadAsStreamAsync())
            {
                var count = 0;
                var buffer = new byte[4096];
                while ((count = await stream.ReadAsync(buffer, 0, buffer.Length)) != 0)
                    /* do nothing */;
            }

            return Ok();
        }
    }
}

Tracked API calls need to be persisted somewhere. In our project, we decided to store the details of each call in a Microsoft Azure Storage Table. For the purposes of unit-testing, we have abstracted this behind the interface and definitions shown below. Implementing this interface is outside the scope of this post and left as an exercise to the reader 🙂

namespace Interop
{
    using System;
    using System.Collections.Generic;
    using System.Threading.Tasks;

    public interface ITableStorage<T> where T : class, new()
    {
        Task<IEnumerable<T>>  EnumRecordsAsync(string query = "");

        Task InsertRecordAsync(T record);
        Task UpdateRecordAsync(T record);
        Task DeleteRecordAsync(T record);
    }
}

Details of a captured REST API call are encapsulated in the following ApiEntry class. It includes important things such as the caller identity and IP address for security purposes, as well as the call duration in order to assess the performance of our API over time.

namespace Storage
{
    using System;
    using System.Collections.Generic;

    using Interop;

    public sealed class ApiEntry
    {
        public ApiEntry()
        {
            TrackingId = Guid.NewGuid();
            CallDateTime = DateTime.UtcNow;
        }
        public Guid TrackingId { get; set; }

        public string CallerIdentity { get; set; }
        public DateTime CallDateTime { get; set; }
        public string CallerAddress { get; set; }

        public string Verb { get; set; }
        public Uri RequestUri { get; set; }
        public IDictionary<String, String[]> RequestHeaders { get; set; }
        public string Request { get; set; }

        public int StatusCode { get; set; }
        public string ReasonPhrase { get; set; }
        public IDictionary<String, String[]> ResponseHeaders { get; set; }
        public string Response { get; set; }

        public TimeSpan CallDuration { get; set; }
    }

    public sealed class ApiTableStorage : ITableStorage<ApiEntry>
    {
       ...
    }
}

Ok, let’s now go back to implementing our Middleware.

A Tracking Owin Middleware

private readonly ITableStorage<ApiEntry> storage_;

public override async Task Invoke(IOwinContext context)
{
    var request = context.Request;
    var response = context.Response;

    // capture details about the caller identity

    var identity =
        request.User != null && request.User.Identity.IsAuthenticated ?
            request.User.Identity.Name :
            "(anonymous)"
            ;

    var record = new ApiEntry
    {
        CallerIdentity = identity,
    };

    // replace the request stream in order to intercept downstream reads

    var requestBuffer = new MemoryStream();
    var requestStream = new ContentStream(requestBuffer, request.Body);
    request.Body = requestStream;

    // replace the response stream in order to intercept downstream writes

    var responseBuffer = new MemoryStream();
    var responseStream = new ContentStream(responseBuffer, response.Body);
    response.Body = responseStream;

    // add the "Http-Tracking-Id" response header

    context.Response.OnSendingHeaders(state =>
    {
        var ctx = state as IOwinContext;
        var resp = ctx.Response;

        // adding the tracking id response header so that the user
        // of the API can correlate the call back to this entry

        resp.Headers.Add("Http-Tracking-Id", new[] { record.TrackingId.ToString("d"), });

    }, context)
    ;

    // invoke the next middleware in the pipeline

    await Next.Invoke(context);

    // rewind the request and response buffers
    // and record their content

    WriteRequestHeaders(request, record);
    record.Request = await WriteContentAsync(requestStream, record.RequestHeaders);

    WriteResponseHeaders(response, record);
    record.Response = await WriteContentAsync(responseStream, record.ResponseHeaders);

    // persist details of the call to durable storage

    await storage_.InsertRecordAsync(record);
}

The Middleware is very simple, and I will walk you through the code.

Remember that this Middleware must be the first in the pipeline, so that is has a chance to intercept the request first and then intercept the response before it returns back to the caller.

So, the Middleware first creates a new instance of the ApiEntry class, which initializes itself with a brand new Tracking Identifier for this request.

Next, the Middleware replaces both the request and response streams taken from the context by custom classes that wrap the streams in order to intercept reads and writes. It gives the Middleware a chance to buffer what’s being read from the request, as well as intercept what’s being written to the response by downstream Middleware components in the pipeline.

When something is first written to in the response Stream, the OnSendingHeaders callback is triggered. The Middleware uses this opportunity to insert the Tracking Identifier in the response headers, so that the client of the call can correlate this invocation with a specific identifier.

The idea being that, should something go wrong with the call, the client will be able to contact Customer support who have all the information available from this call recorded in durable storage, including the reason for the error and, possibly, the underlying exception and its call stack as well.

Then, the Middleware invokes the next Middleware component in the pipeline, thus triggerring the sequence of the calls to the ultimate REST API.

Finally, the Middleware rewinds the local Stream buffers and records all the details of this call to durable storage.

The following helper methods are used in the Middleware implementation, shown here for completeness:

private static void WriteRequestHeaders(IOwinRequest request, HttpEntry record)
{
    record.Verb = request.Method;
    record.RequestUri = request.Uri;
    record.RequestHeaders = request.Headers;
}

private static void WriteResponseHeaders(IOwinResponse response, HttpEntry record)
{
    record.StatusCode = response.StatusCode;
    record.ReasonPhrase = response.ReasonPhrase;
    record.ResponseHeaders = response.Headers;
}

private static async Task<string> WriteContentAsync(ContentStream stream, IDictionary<string, string[]> headers)
{
    const string ContentType = "Content-Type";

    var contentType = 
        headers.ContainsKey(ContentType) ?
        headers[ContentType][0] :
        null
        ;

    return await stream.ReadContentAsync(contentType);
}

We’re using a custom Stream class that helps us intercept the reads and writes as well as provide some helper code to read the streams as text. If the HTTP content-type response header denotes a “string” content, for instance, application/json or text/plain, the Stream class attempts to identify the encoding and read the Stream as text. Otherwise, it might be binary so no attempt to read the Stream is made.

public class ContentStream : Stream
{
    protected readonly Stream buffer_;
    protected readonly Stream stream_;

    private long contentLength_ = 0L;
        
    public ContentStream(Stream buffer, Stream stream)
    {
        buffer_ = buffer;
        stream_ = stream;
    }

    /// <summary>
    /// Returns the recorded length of the underlying stream.
    /// </summary>
    public virtual long ContentLength
    {
        get { return contentLength_; }
    }

    public async Task<String> ReadContentAsync(string contentType, long maxCount)
    {
        if (!IsTextContentType(contentType))
        {
            contentType = String.IsNullOrEmpty(contentType) ? "N/A" : contentType;
            return String.Format("{0} [{1} bytes]", contentType, ContentLength);
        }

        buffer_.Seek(0, SeekOrigin.Begin);

        var length = Math.Min(ContentLength, maxCount);

        var buffer = new byte[length];
        var count = await buffer_.ReadAsync(buffer, 0, buffer.Length);

        return
            GetEncoding(contentType)
            .GetString(buffer, 0, count)
            ;
    }

    protected void WriteContent(byte[] buffer, int offset, int count)
    {
        buffer_.Write(buffer, offset, count);
    }

    #region Implementation

    private static bool IsTextContentType(string contentType)
    {
        if (contentType == null)
            return false;

        var isTextContentType =
            contentType.StartsWith("application/json") ||
            contentType.StartsWith("application/xml") ||
            contentType.StartsWith("text/")
            ;
        return isTextContentType;
    }

    private static Encoding GetEncoding(string contentType)
    {
        var charset = "utf-8";
        var regex = new Regex(@";\s*charset=(?<charset>[^\s;]+)");
        var match = regex.Match(contentType);
        if (match.Success)
            charset = match.Groups["charset"].Value;

        try
        {
            return Encoding.GetEncoding(charset);
        }
        catch (ArgumentException e)
        {
            return Encoding.UTF8;
        }
    }

    #endregion

    #region System.IO.Stream Overrides

    ...

    public override int Read(byte[] buffer, int offset, int count)
    {
        // read content from the request stream

        count = stream_.Read(buffer, offset, count);
        contentLength_ += count;

        // record the read content into our temporary buffer

        if (count != 0)
            WriteContent(buffer, offset, count);

        return count;
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        // store the bytes into our local stream

        WriteContent(buffer, 0, count);

        // write the bytes to the response stream
        // and record the actual number of bytes written

        stream_.Write(buffer, offset, count);
        contentLength_ += count;
    }

    #endregion
       
    #region IDisposable Implementation

    protected override void Dispose(bool disposing)
    {
        buffer_.Dispose();

        // not disposing the stream_ member
        // which is owned by the Owin infrastructure
    }

    #endregion
}

That’s it. As you can see, it’s very easy to implement a custom Middleware in order to intercept the request and/or the response. A couple of points of interest in this post have shown you how you can replace the response stream, how you can insert your own headers in the response, etc. A further point of note is that by using a custom Stream-based class, the processing happens at the same time as the downstream Middleware components are reading from the request and/or writing to the response streams.

In a real production code, I would not use instances of the MemoryStream class for my temporary buffers because I’m concerned about having the whole content of both the request and the response live in memory for the duration of the call. In my project, I’m using an instance of the VirtualStream class, available as a sample from BizTalk Server, in order to make sure that not too much memory is taken by the streams.

Code available on GitHub

I have set a GitHub repository hosting an updated and more complete version of this code.

Please, feel free to have a look.