unit testing - xUnit Async Test Not Working Properly

Question

We have been using xUnit Framework in our project as test framework since the begining. Currently there are 2200+ unit tests in project and everything seems fine.

But yesterday i decided to run unit tests at CI builds and nightly builds. Fighting with team build controller to run xunit tests for 1-2 hours i succeed to run tests. But there was a problem, there 22 warning abouts tests like below, - Xunit.Sdk.EqualException: Assert.Equal() Failure or - System.ArgumentNullException: Value cannot be null.

After some more research i realized these tests have to be failed but seems passed because all of these tests are marked with "async" keyword.

Actually there shouldn't be any error, because xUnit supports async tests regarding these posts http://bradwilson.typepad.com/blog/2012/01/xunit19.html http://sravi-kiran.blogspot.com.tr/2012/11/UnitTestingAsynchronousMethodsUsingMSTestAndXUnit.html Howerver reality is slighter difference for me. Yes xUnit runs without any problem but doesn't test properly.

Here are two little methods.

public async Task<string> AsyncTestMethod() {
    throw new Exception("Test");
}
public string NormalTestMethod() {
    throw new Exception("Test");
}

As you can see only difference, the first method defined as "async" And here are two tests for these methods.

  [Fact]
    public async void XunitTestMethod_Async() {
        Program p = new Program();
        string result = await p.AsyncTestMethod();
        Assert.Equal("Ok", result);
    }
        [Fact]
    public  void XunitTestMethod_Normal() {
        Program p = new Program();
        string result =  p.NormalTestMethod();
        Assert.Equal("Ok", result);
    }

Both of the original methods throws exception,so i think both of the tests should fail but the result is different. You can see the results:

XunitTestMethod_Async test pass but XunitTestMethod_Normal fails. Throwing an exception is not the case, you can change the method content whatever you want AsyncTestMethod always passes.

Here is the example solution: https://github.com/bahadirarslan/AsyncTestWithxUnit

I might misunderstood or think wrong but now this behaviour causes lots of pain for me

Hope you can enlight me.

PS: I created an issue at xUnit github page, but i couldn't be sure that is this caused from me or xUnit. So i decided to ask here too. Issue: https://github.com/xunit/xunit/issues/96

Answer 1

The problem is that async void method does not return a Task object which xUnit could possibly examine for exceptions. xUnit cannot even track the completion of a particular async void method, as opposed to an async Task method.

Moreover, async void methods do not throw exceptions on the same stack frame, even for exceptions thrown from the synchronous part of the method. Rather, exceptions are propagated to the current synchronization context via SynchronizationContext.Post (in case SynchronizationContext.Current != null), or otherwise via ThreadPool.QueueUserWorkItem. This behavior is different from async Task methods, more details. It's possible to track the total number of the pending async void methods (via SynchronizationContext.OperationStarted/OperationCompleted), but not individual methods.

So, changing the XunitTestMethod_Async method's signature from async void to async Task should solve the problem.

Updated, it's 2019 and as @maracuja-juice points out, both async void tests now fail, as they original should have behaved. xUnit team has implemented AsyncTestSyncContext, their special flavor of synchronization context, an instance of which they use to individually track each async void call and capture any exceptions.