Testing Azure Service Bus Handlers using the  Azure.Messaging.ServiceBus Client Library

In April 2020, Microsoft released a new client library for Azure Service Bus called Azure.Messaging.ServiceBus. This new library is based on the open Advanced Message Queuing Protocol (AMQP) standard. Microsoft outlines that "AMQP enables you to build cross-platform, hybrid applications using a vendor-neutral and implementation-neutral, open standard protocol. You can construct applications using components that are built using different languages and frameworks, and that run on different operating systems. All these components can connect to Service Bus and seamlessly exchange structured business messages efficiently and at full fidelity."

The class library documentation hints at this library being testable! One of the frustrations of using previous Service Bus client libraries was that absolutely everything was sealed or protected, meaning the only way to test client code was to wrap up and abstract away the service bus infrastructure. The situation might have been helped by a local emulator, but despite being a highly requested and up-voted feature request, an emulator was never officially implemented. So, any improvements to the testability of Service Bus client code are very welcomed.

The test support is a few classes that are public and not sealed, a sprinkling of overridable methods and a factory class for creating models.

There are no official samples yet and very few (if any) blogs covering how to test. But after a bit of trial and error, we managed to get some tests working that we’re happy with, hence sharing here so that others could benefit. Here we cover testing the very basic handling of messages, although it should also be a useful starting point for testing more advanced features.

Component tests

At ClearBank, we’re favouring 'component tests' over reams of unit tests – which I see as a kind of internal integration test. We run a test host, using real concrete objects, except for anything that touches the network – these objects are mocked. We test as much of the service as possible (i.e. including the Startup class and the Program class if possible). The tests include happy and sad paths, enough to give sufficiently high line and branch coverage of the tested services. These types of tests take longer to set up than pure unit tests making Test Driven Development (TDD) a bit harder to start with, but they run locally, much faster and are less brittle than integration/end-to-end tests. This gives the same confidence that the business logic is proven in a smaller number of tests than traditional unit tests. Of course, you still need a couple of integration tests, but these need only prove that the integration is working, rather than all the paths through the logic.

The main requirement when testing a service bus message handler is to be able to create a ServiceBusReceivedMessage. This class doesn't have a public constructor, but there is ServiceBusModelFactory class with a ServiceBusReceivedMessage method especially for the purpose! This needs to be at the heart of our test approach.

When we create the artificial ServiceBusReceivedMessage it needs to be packaged up in a ProcessMessageEventArgs however, if we want to spy on what happens to the message, we can inherit from this class and override CompleteMessageAsyncDeadLetterMessageAsync and AbandonMessageAsync methods and set flags accordingly. We called this class TestableProcessMessageEventArgs.

TestableProcessMessageEventArgs class

{
    public bool WasCompleted{ get; private set; }
    public bool WasDeadLettered{ get; private set; }
    public DateTime Created{ get; }

    public TestableProcessMessageEventArgs(ServiceBusReceivedMessage message) : base(message, null, CancellationToken.None)
    {
        Created = DateTime.Now;
    }

    public override Task CompleteMessageAsync(ServiceBusReceivedMessage message,
        CancellationToken cancellationToken = new CancellationToken())
    {
        WasCompleted = true;
        return Task.CompletedTask;
    }

    public override Task DeadLetterMessageAsync(ServiceBusReceivedMessage message, string deadLetterReason,
        string deadLetterErrorDescription = null, CancellationToken cancellationToken = new())
    {
        WasDeadLettered = true;
        return Task.CompletedTask;
    }

    public override Task AbandonMessageAsync(ServiceBusReceivedMessage message, IDictionary<string, object> propertiesToModify = null,
        CancellationToken cancellationToken = new CancellationToken())
    {
        return Task.CompletedTask;
    }
}```

The next thing we need to mock is the ServiceBusProcessor class. This is returned by the ServiceBusClient and communicates with the service bus, passing messages to the handler for a specific queue or topic. We could try to use Moq but it’s easier to inherit from ServiceBusProcessor and override its StartProcessingAsync() method. This stops it from realising that it isn't connected to a real service bus. We called this class TestableServiceBusProcessor.

We can then call base.OnProcessMessageAsync(args) on the TestableServiceBusProcessor passing in a an artificial message created using the ServiceBusModelFactory.ServiceBusReceivedMessage() mentioned above. This will present the artificial message to the handler, as if it were connected to a real service bus receiving a real message.

We can add a couple of helpful features to the TestableServiceBusProcessor such as a collection of TestableProcessMessageEventArgs called MessageDeliveryAttempts to assert against and a generic method for sending messages. We can also simulate retries in order to test sad paths and/or exponential back off policies etc.

TestableServiceBusProcessor class

{
    public List<TestableProcessMessageEventArgs> MessageDeliveryAttempts = new();

    public async Task SendMessageWithRetries<T>(T payload, int maxDeliveryCount = 5)
    {
        for (var attempt = 1; attempt <= maxDeliveryCount; attempt++)
        {
            // Don't send retry if the message was sent already and completed
            if (MessageDeliveryAttempts.Any() && MessageDeliveryAttempts.Last().WasCompleted)
                return;

            await SendMessage(payload, attempt);

            // Simulate the message being deadlettered if max delivery count is hit
            if (attempt == maxDeliveryCount)
                MessageDeliveryAttempts.Last().WasDeadLettered = true;

        }
    }

    public async Task SendMessage<T>(T payload, int attempt = 1)
    {
        var args = CreateMessageArgs(payload, attempt);
        MessageDeliveryAttempts.Add((TestableProcessMessageEventArgs)args);
        await base.OnProcessMessageAsync(args);
    }

    public ProcessMessageEventArgs CreateMessageArgs<T>(T payload, int deliveryCount = 1)
    {
        var payloadJson = JsonSerializer.Serialize(payload);

        var message = ServiceBusModelFactory.ServiceBusReceivedMessage(
            body: BinaryData.FromString(payloadJson),
            deliveryCount: deliveryCount);

        var args = new TestableProcessMessageEventArgs(message);

        return args;
    }

    public override async Task StartProcessingAsync(CancellationToken cancellationToken = default)
    {
    }
}```

How to use them

To use these classes in a component test, we can use the TestHost from the Microsoft.AspNetCore.TestHost package. This enables us to run a service in memory, calling the ConfigureServices method as normal, but then overriding some of the configurations (i.e. the ones that touch the network that we need to mock). Consider the following code:

{
    private readonly IHost _server;
    public Mock<ServiceBusSender> MockTestQueue2Sender { get; } = new();
    public TestableServiceBusProcessor TestableQueue1MessageProcessor { get; } = new();
    public Mock<IWidget> MockWidgetService { get; } = new();
    public IServiceProvider Services => _server.Services;
    public int NumberOfSimulatedServiceBusMessageRetries = 5;
    public int InitialRetryDelayForServiceBusMessageRetriesInMs = 100;

    public TestHost()
    {
        var builder = new HostBuilder()
            .ConfigureWebHost(webHost =>
            {
                webHost.UseTestServer()
                    .ConfigureServices((context, services) => Program.ConfigureHost(services, context.Configuration))
                    .ConfigureTestServices((services) =>
                    {
                        var client = new Mock<ServiceBusClient>();

                        client.Setup(t => t.CreateSender(It.Is < string > (s => s == "testQueue2")))
                            .Returns(MockTestQueue2Sender.Object);

                        client.Setup(t => t.CreateProcessor(It.Is<string>(s => s == $"testQueue1")))
                            .Returns(TestableQueue1MessageProcessor);

                        services.AddSingleton(client.Object);

                        // register other test services here...
                        services.AddSingleton(MockWidgetService.Object);

                    }).Configure(app => { });
            });

        _server = builder.Build();
        _server.StartAsync();
    }

    public void Dispose()
    {
        _server?.StopAsync().GetAwaiter().GetResult();
        _server?.Dispose();
    }
}```

We're using Moq to create a Mock ServiceBusClient. This will be registered into the DI container and when it's CreateProcessor() method is called with the correct queue name, an instance of our TestableServiceBusProcessor will be returned. We are also setting up a Mock ServiceBusSender which we can use later to Assert whether messages were sent to a different queue. We have a Mock IWidget service, which our message handler calls and we can use to throw exceptions or simulate processing delays etc.

The actual tests

The following code shows the actual tests. We decided to split the setup, execution and assertion into a set of helpers named GivenWhen and Then. This allows for a nice fluent interface for the tests and makes them nice and readable.

public void a_message_sent_to_the_queue_is_handled_and_completed()
{
    using var host = new TestHost();

    Given.OnThe(host)
        .ATestPayloadIsGenerated(out var testPayload);

    When.OnThe(host)
        .AMessageIsSentToTestQueue1(testPayload);

    Then.OnThe(host)
        .TheWidgetServiceWasCalled(times: 1)
        .And().AMessageWasSentToTestQueue2()
        .And().TheMessageWasCompleted();
}

[Fact]
public void a_message_sent_to_the_queue_is_handled_and_when_permanentException_is_thrown_is_dead_lettered()
{
    using var host = new TestHost();

    Given.OnThe(host)
        .ATestPayloadIsGenerated(out var testPayload)
        .And().TheWidgetWillThrowAPermanentException();

    When.OnThe(host)
        .AMessageIsSentToTestQueue1(testPayload);

    Then.OnThe(host)
        .AMessageWasSentToTestQueue2(times: 0)
        .And().TheMessageWasDeadLettered();
}

[Fact]
public void a_message_sent_to_the_queue_is_handled_and_when_transientException_is_thrown_is_retried_and_is_eventually_completed()
{
    using var host = new TestHost();

    Given.OnThe(host)
        .ATestPayloadIsGenerated(out var testPayload)
        .And().TheWidgetWillThrowANumberOfTransientExceptions(times:3);

    When.OnThe(host)
        .AMessageIsSentToTestQueue1(testPayload, simulateRetries: true);

    Then.OnThe(host)
        .TheMessageWasRetried(times:4)
        .And().TheWidgetServiceWasCalled(times:4)
        .And().TheRetriedMessagesHadIncreasingDelays()
        .And().AMessageWasSentToTestQueue2(times:1)
        .And().TheMessageWasCompleted();
}```

Given, When and Then test helpers

Here is the code for the When class which simulates the message appearing on the queue:

{
    private readonly TestHost _host;

    public When(TestHost host)
    {
        _host = host;
    }

    public static When OnThe(TestHost host) => new(host);
    public When And() => this;

    public When AMessageIsSentToTestQueue1(string testPayload, bool simulateRetries = false)
    {
        var payload = new TestQueueMessage(testPayload);

        if (simulateRetries)
            _host.TestableQueue1MessageProcessor.SendMessageWithRetries(payload, _host.NumberOfSimulatedServiceBusMessageRetries).GetAwaiter().GetResult();
        else
            _host.TestableQueue1MessageProcessor.SendMessage(payload).GetAwaiter().GetResult();

        return this;
    }
}```

And here is the code for the Then class which demonstrates how to use the TestableServiceBusProcessor and its collection of MessageDeliveryAttempts, each of which contains a TestableProcessMessageEventArgs that be can used to make assertions. We are using the FluentAssertions package.

{
    private readonly TestHost _host;

    public Then(TestHost host)
    {
        _host = host;
    }

    public static Then OnThe(TestHost host) => new(host);
    public Then And() => this;

    public Then TheMessageWasCompleted()
    {
        _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts.Last().WasCompleted.Should().BeTrue();
        return this;
    }

    public Then TheMessageWasDeadLettered()
    {
        _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts.Last().WasDeadLettered.Should().BeTrue();
        return this;
    }

    public Then TheMessageWasRetried(int times)
    {
        _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts.Count.Should().Be(times);
        return this;
    }

    public Then TheWidgetServiceWasCalled(int times) 
    {
        _host.MockWidgetService.Verify(x => x.DoSomething(It.IsAny<string>()), Times.Exactly(times));
        return this;
    }

    public Then AMessageWasSentToTestQueue2(int times = 1)
    {
        _host.MockTestQueue2Sender.Verify(x => x.SendMessageAsync(It.IsAny<ServiceBusMessage>(), It.IsAny<CancellationToken>()), Times.Exactly(times));
        return this;
    }

    public Then TheRetriedMessagesHadIncreasingDelays()
    {
        var delays = new List<TimeSpan>();
        for (var i = 0; i < _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts.Count - 1; i++)
        {
            delays.Add(_host.TestableQueue1MessageProcessor.MessageDeliveryAttempts[i + 1].Created -
                        _host.TestableQueue1MessageProcessor.MessageDeliveryAttempts[i].Created);
        }

        for (var i = 0; i < delays.Count - 1; i++)
        {
            Assert.True(delays[i + 1] > delays[i], $"the interval of retry{i+1}({delays[i+1].TotalMilliseconds}ms) was not larger than the previous retry({delays[i].TotalMilliseconds}ms)");
        }

        return this;
    }
}```

So that's how we're testing message handling code that uses the Azure.Messaging.Servicebus client library. If you would like to learn more, get in touch with Andrew Poole.

All the code can be found in this GitHub repo. Image shared from rarehistoricalphotos.com.

Andrew Poole

Senior Software Engineer, ClearBank