Intelli-Test (Part 2)

March 25, 2016

I recently posted an article which morphed into a discovery of the Intelli-Test feature in VS2015.

My initial findings were relating to creating a basic intelli-test, and then having that create a new unit test for me. However, once you’ve created an intelli-test, you can modify it; here’s the original intelli-test that was created for me:



        [PexGenericArguments(typeof(int))]
        [PexMethod]
        internal void ClearClassTest<T>(T classToClear)
        {
            Program.ClearClass<T>(classToClear);
            // TODO: add assertions to method ProgramTest.ClearClassTest(!!0)
        }

When this creates a test, it looks like this:




[TestMethod]
[PexGeneratedBy(typeof(ProgramTest))]
public void ClearClassTest861()
{
    this.ClearClassTest<int>(0);
}

So, I tried adding some additional parameters:




    class TestClass
    {
        public string Test1 { get; set; }
        public string Test2 { get; set; }
    }

    /// <summary>This class contains parameterized unit tests for Program</summary>
    [PexClass(typeof(Program))]
    [PexAllowedExceptionFromTypeUnderTest(typeof(InvalidOperationException))]
    [PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException), AcceptExceptionSubtypes = true)]
    [TestClass]
    public partial class ProgramTest
    {
        /// <summary>Test stub for ClearClass(!!0)</summary>
        [PexGenericArguments(typeof(int))]
        [PexGenericArguments(typeof(string))]
        [PexGenericArguments(typeof(float))]
        [PexGenericArguments(typeof(TestClass))]
        [PexMethod]
        internal void ClearClassTest<T>(T classToClear)
        {
            Program.ClearClass<T>(classToClear);
            // TODO: add assertions to method ProgramTest.ClearClassTest(!!0)
        }
    }

Just re-select “Run intelli-test” and it updates the ProgramTest.ClearClassTest.g.cs, generating 6 new tests. To be honest, this was a bit disappointing. I had expected an “intelligent” test - that is, one that tests several outcomes. To simplify what was happening, I tried creating a simple method:




        public static int TestAdd(int num1, int num2)
        {
            return num1 + num2;
        }

Creating the test for this resulted in this:




        [PexMethod]
        internal int TestAddTest(int num1, int num2)
        {
            int result = Program.TestAdd(num1, num2);
            return result;
            // TODO: add assertions to method ProgramTest.TestAddTest(Int32, Int32)
        }

And then to:




[TestMethod]
[PexGeneratedBy(typeof(ProgramTest))]
public void TestAddTest989()
{
    int i;
    i = this.TestAddTest(0, 0);
    Assert.AreEqual<int>(0, i);
}

So, then I considered what this was actually doing. The purpose of it seems to be to execute code; that is, code coverage; so what happens if I create a method like this:




        public static int TestAdd(int num1, int num2)
        {
            if (num1 > num2)
                return num1 + num2;
            else
                return num2 - num1;
        }

And finally, you see the usefulness of this technology. If it only creates a single test, passing in 0,0 then it only tests one code path. The minimum test to cover all code paths is 0,0 and 1,0. That’s exactly what it does; the generated test looks the same:




        /// <summary>Test stub for TestAdd(Int32, Int32)</summary>
        [PexMethod]
        internal int TestAddTest(int num1, int num2)
        {
            int result = Program.TestAdd(num1, num2);
            return result;
            // TODO: add assertions to method ProgramTest.TestAddTest(Int32, Int32)
        }

But the run intelli-test creates two methods:




[TestMethod]
[PexGeneratedBy(typeof(ProgramTest))]
public void TestAddTest989()
{
    int i;
    i = this.TestAddTest(0, 0);
    Assert.AreEqual<int>(0, i);
}

[TestMethod]
[PexGeneratedBy(typeof(ProgramTest))]
public void TestAddTest365()
{
    int i;
    i = this.TestAddTest(1, 0);
    Assert.AreEqual<int>(1, i);
}

In an effort to confuse the system, I changed the base function:




        public static int TestAdd(int num1, int num2)
        {
            num1++;

            if (num1 == num2)
            {
                throw new Exception("cannot be the same");
            }

            if (num1 > num2)
                return num1 + num2;
            else
                return num2 - num1;
            
        }

And re-run the test; it resulted in 3 tests:

Not sure where the numbers came from, but this tests every code path. The exception results in a fail, but you can mark that as a pass by simply right clicking and selecting “Allow”; which changes the test to look like this:




[TestMethod]
[PexGeneratedBy(typeof(ProgramTest))]
[ExpectedException(typeof(Exception))]
public void TestAddTestThrowsException26601()
{
    int i;
    i = this.TestAddTest(502, 503);
}

Summary

This is basically the opposite of the modern testing philosophy, which is test interactions and behaviour; what this does is establish every code path in a function and run it; so it should establish that the code doesn’t crash, but it obviously can’t establish that it does what it’s intended to do, nor can it establish whether it is sensible to pass, for example, a null value. Having said that, if an object can be null, and there’s no defensive code path to deal with it being passed null, then this will be pointed out.

In general, this is intended for people dealing with, and refactoring, legacy code so that they can adopt the “Stangling Vines” development pattern in order to be sure that the code still does what it did before.