fd Blog

Daniel Hilgarth on software development

Unit-testing With AutoFixture, NCrunch and Global State

Trying to unit-test code that relies on global state can be a frustrating experience. Global state is shared among tests, making the tests no longer independent. This can lead to subtle bugs in the test suite, e.g. a scenario where tests seem to fail randomly when executed in bigger batches, but pass when executed alone. Tracking down - and eliminating - the state sharing can be both time consuming and frustrating.

In certain scenarios, the effect might be less subtle, as is the case in the scenario I encountered while writing my first unit-tests for a new mobile app that uses MvvmCross. Tracking it down, however, was no less time consuming or frustrating.

My test suite uses AutoData theories, a feature of AutoFixture that can create anonymous objects for xUnit’s theories. That feature works in a way that it creates a new instance of AutoFixture per test and configures it according to the requirements of the test project using customizations.
If a test depends on global state and that global state should be created by AutoFixture, a customization is the place to initialize that global state.
The MvvmCrossTestSetupCustomization from the previous post is an example of such a customization.

This works - unless you are using NCrunch.
NCrunch is a Visual Studio Add-in that runs your tests automatically, whenever you change your code. You don’t have to manually start a build or test run.

NCrunch analyses the tests and - besides other things - it analyses the attributes you placed on your test methods.
This analysis results in the creation of an instance of each attribute on your test.
This includes the AutoData attribute which creates a new instance of AutoFixture and sets the global state.
xUnit also creates instances of the attributes on the tests.

For each test, this leads to two instances of AutoFixture to be created. One in the attribute instance that will later be used by xUnit to provide the values for our test and the other will be created in the attribute instance that is created by NCrunch’s analysis of the test.

Normally, that isn’t a problem, but as soon as such a customization sets global state, it does get problematic:

  • The attribute instance from xUnit sets the global state with data created by AutoFixture instance A
  • The attribute instance from NCrunch sets the global state with data created by AutoFixture instance B

Depending on the order those instances get created, either the data from AutoFixture instance A will be overwritten by the data from AutoFixture instance B or vice versa.
If the instance from NCrunch is created first we would have no problem: The global state would contain data from AutoFixture instance A and the data that is supplied to the test method would also come from instance A. In effect, the System Under Test that uses the global state operates on the same data as the test method.

Unfortunately, the order is switched:
The instance from xUnit gets created first and only afterwards the instance from NCrunch gets created, overwriting the global state. In effect, the SUT and the test method now operate on different data.

The solution seems simple: Don’t execute the customization a second time by introducing a static field.
Unfortunately, that doesn’t work.
Remember: Each test gets its own instance of AutoFixture. But the static field would result in the customization to be executed once - for the whole test suite! So all our tests - except the first one - would be executed without our customization applied.

To actually solve the problem, we need to resort to a bit of a hack:
Don’t execute our customization when we have been created by NCrunch.
This is a hack because we have to rely on NCrunch implementation details, more precisely: on the call stack.

If we can accept that, the solution is rather simple:

  1. Create a decorator customization that executes the decorated customization only if the call stack indicates that we have been called by xUnit.net and net by NCrunch.
  2. Decorate our original customization with it

Decorator

public class DisabledWhenCreatedByNCrunchCustomization : ICustomization
{
    private readonly ICustomization _decorated;

    public DisabledWhenCreatedByNCrunchCustomization(ICustomization decorated)
    {
        if (NumberOfItemsUntilNamespace(stack, "xunit") > NumberOfItemsUntilNamespace(stack, "ncrunch"))
            return;
        _decorated = decorated;
    }

    public void Customize(IFixture fixture)
    {
        if (_decorated != null)
            _decorated.Customize(fixture);
    }

    private static MethodBase[] GetStackFrameMethods()
    {
        var stackFrames = new StackTrace().GetFrames();
        if (stackFrames == null)
            return new MethodBase[0];
        return stackFrames.Select(x => x.GetMethod()).Where(x => x != null).ToArray();
    }

    private static int NumberOfItemsUntilNamespace(
        IEnumerable<MethodBase> stack, string beginningOfNamespace)
    {
        return
            stack.TakeWhile(
                x => !x.DeclaringType.Namespace
                       .StartsWith(beginningOfNamespace, StringComparison.OrdinalIgnoreCase))
                 .Count();
    }
}

Updated registration

public class TestCustomization : CompositeCustomization
{
    public TestCustomization()
        : base(new AutoNSubstituteCustomization(), new AnonymousCreationCustomization(),
/* here --> */  new DisabledWhenCreatedByNCrunchCustomization(new MvvmCrossTestSetupCustomization()))
    {
    }

    private class AnonymousCreationCustomization : ICustomization
    {
        public void Customize(IFixture fixture)
        {
            // ...
        }
    }
}

Final words

Please note that this problem or its solution is not constrained to MvvmCross projects. The global state could just as well have been a static property.

Comments