c# - Is Nunit ExpectedException Working Correctly?

Question

I started having problems earlier when setting up some command line scripts to test my NUnit suite of tests for a project I am working on.

I noticed that when running the tests in Visual Studio or Xamarin that the results are what I expected, but when running from the command line with the nunit-console (mac v2.4.8) it doesn't fail some of the tests.

All the of the tests that are failing that don't fail in the console use the [ExpectedException] attribute (both with a type and in general). When changing to use Assert.Throws<> it works correctly in both the IDEs and the command line.

Is this a bug with Nunit or the particular version/platform I have?

Answer 1

It is a bug in unit-console 2.4.8 and has nothing to due with VS test runner as one commenter suggested. I use 3.x on the cmd-line to work around it by doing a local nuget install as I add this into my Makefile and can do a 'make test'.

Testing via 2.4.8 (and exposing the bug/issue):

nunit-console -nologo -labels except.dll

***** except.Test.ExpectedException
***** except.Test.ExpectedNotSystemException
***** except.Test.ExpectedNotTypeOfSystemException
***** except.Test.NotExpectedException

Tests run: 4, Failures: 1, Not run: 0, Time: 0.106 seconds

Test Case Failures:
1) except.Test.NotExpectedException : System.Exception : Stackoverflow
at except.Test.NotExpectedException () [0x00006] in /Users/sushi/code/XamTests/except/except/Test.cs:33
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in /private/tmp/source-mono-mac-4.2.0-branch-c6sr1/bockbuild-mono-4.2.0-branch/profiles/mono-mac-xamarin/build-root/mono-4.2.2/mcs/class/corlib/System.Reflection/MonoMethod.cs:295

Local 3.0.1 install and test (works correctly):

nuget install nunit.runners

./NUnit.Console.3.0.1/tools/nunit3-console.exe except.dll

NUnit Console Runner 3.0.5813
~~~  
Errors and Failures

1) Failed : except.Test
One or more child tests had errors
2) Failed : except.Test.ExpectedNotSystemException
An unexpected exception type was thrown
3) Failed : except.Test.ExpectedNotTypeOfSystemException
An unexpected exception type was thrown
4) Error : except.Test.NotExpectedException

Test Run Summary
    Overall result: Failed
   Tests run: 4, Passed: 1, Errors: 1, Failures: 2, Inconclusive: 0
     Not run: 0, Invalid: 0, Ignored: 0, Explicit: 0, Skipped: 0
  Start time: 2016-01-26 23:09:56Z
    End time: 2016-01-26 23:09:56Z
    Duration: 0.117 seconds

Test Case:

using NUnit.Framework;
using System;

namespace except
{
    [TestFixture ()]
    public class Test
    {
        [Test ()]
        [ExpectedException]
        public void ExpectedException ()
        {
            throw new Exception ("Stackoverflow");
        }

        [Test ()]
        [ExpectedException("System.DivideByZeroException")]
        public void ExpectedNotSystemException ()
        {
            throw new Exception ("Stackoverflow");
        }

        [Test ()]
        [ExpectedException(typeof(DivideByZeroException))]
        public void ExpectedNotTypeOfSystemException ()
        {
            throw new Exception ("Stackoverflow");
        }

        [Test ()]
        public void NotExpectedException ()
        {
            throw new Exception ("Stackoverflow");
        }
    }
}