Assert.Count in MbUnit v3.2

Assert.Count is a last-minute addition made to Gallio/MbUnit v3.2 just before passing to RC. Be sure to read the nice article that Vadim Kreynin has written about that new convenient assertion.

Gallio & MbUnit 3.2 RC

Graham has unleashed today Gallio v3.2 RC in the wild:

The latest build of Gallio & MbUnit 3.2 (build 517) is considered a Release Candidate, you can find it here. Release notes are on the wiki. This means no new features will be added, but obviously any critical bugs will be fixed before the official release.

We're aware that it's been a long time since the last release, and we're hoping to push more frequent smaller releases over the coming months.

Please download it and let us know what you think!

Gallio v3.2 + ReSharper 5.1

As you may know, Graham Hay is one of the major contributors of the Gallio project. He has committed recently an amazing amount of work to make Gallio support the latest version of JetBrains ReSharper.

The version 5.1 of R# has not been officially released yet; but you can run your tests with Gallio and that very popular Visual Studio extension already.

Download here the latest build of the Gallio bundle.

Niklas Dahlman talks about advanced features of MbUnit v3

It seems that I had totally missed that great presentation by Niklas Dahlman about some advanced features of MbUnit v3. The session was recorded in November 2009 during the Øredev conference.

In particular I like very much the part about the contract verifiers. Niklas shows a couple of very nice examples.

Hash Code Acceptance Contract Verifier in MbUnit v3

Designing a good hash code generation algorithm is a black art. There is no formal and objective way to determine what is the best algorithm for a given scenario. In fact, most of the developers do not know how exactly to implement efficiently hash functions. As you know, hash functions for a given type are implemented in .NET by overriding the method Object.GetHashCode(). Most of us (including myself, I must admit) usually shake up and down the values by shifting and xoring them randomly without knowing exactly if the result will be good enough. It’s even worse: if the implementation is poor, your application will continue to work because a bad hash function does "only" affect performance. So you might even not be able to notice it immediately.

In the other hand, testing a hash function is cumbersome. A good hash function should have the following properties:

• Low probability of collision; meaning that the odds to get two values that produces the same hash code should be as low as possible.
• Hash codes should be distributed uniformly.
• It should achieve "avalanche" by generating hash values wildly different if even a single bit is different in the input key.

Writing unit tests for a hash generator is therefore not a trivial task. Fortunately MbUnit comes to rescue. The soon released v3.2 provides a new contract verifier named Hash Code Acceptance Contract.

The contract verifier adds two child tests to the test fixture. They evaluate the probability of collision and the uniform distribution goodness-of-fit. At this time, the avalanche test is not supported (perhaps in a future release?)

Consider the sample type below. It implements an awful hash function of the additive kind. Let's imagine that according to some imaginary specifications, value is a number between 0 and 9, and dayOfWeek is... well, I just let you find out :)

public class Foo
{
  private readonly int value;
  private readonly DayOfWeek dayOfWeek;

  public Foo(int value, DayOfWeek dayOfWeek)
  {
    this.value = value;
    this.dayOfWeek = dayOfWeek;
  }

  public override int GetHashCode()
  {
    return value + (int)dayOfWeek;
  }
}

Now let's use the contract verifier to evaluate our poor implementation.

[TestFixture]
public class FooTest
{
  [VerifyContract]
  public readonly IContract HashCodeAcceptanceTests = new HashCodeAcceptanceContract<Foo>()
  {
    CollisionProbabilityLimit = CollisionProbability.Low,
    UniformDistributionQuality = UniformDistributionQuality.Good,
    DistinctInstances = DataGenerators.Join(
                          Enumerable.Range(0, 10),
                          Enum.GetValues(typeof(DayOfWeek)).Cast<DayOfWeek>())
                          .Select(o => new Foo(o.First, o.Second))
  };
}

CollisionProbabilityLimit and UniformDistributionQuality are the expected confidence levels. These are probability values between 0 (good) and 1 (bad), but we use here some handy predefined constants to improve readability (more details here and here). You may want to let the default values of 5% if you are not sure.

DistinctInstances must be fed with an enumeration of distinct Foo instances. It's important to understand that the entire evaluation of the hash function is based on the statistical population you decide to provide. Therefore the confidence in the test results will be as good as (or as bad as, for the pessimists) the quality and the representativeness of that population. Two types of scenarios are possible. Either the range of possibilities is finite and reasonably small, or the number of possible distinct instances is nearly infinite or insanely large. In the first case, it's better to provide to the contract verifier all the possible values. Thus you get an exact and complete stochastic evaluation of your hash function. In the second case, you will need to carefully select a representative subset of all the possible values.

In both cases you can feed the contract verifier by using a static method which returns an enumeration.

// ...
DistinctInstances = GetDistinctInstances()
// ...

private static IEnumerable<Foo> GetDistinctInstances()
{
  // ...
}

Or to provide an existing catalog if they can be retrieved from an external data source.

// ...
DistinctInstances = Foo.GetThemAll()
// ...

You may also use the powerful MbUnit data generation framework to easily combine and generate random or sequential values.

// ...
DistinctInstances = DataGenerators.Join(
                      Enumerable.Range(0, 10), 
                      Enum.GetValues(typeof(DayOfWeek)).Cast<DayOfWeek>())
                      .Select(o => new Foo(o.First, o.Second))
// ...

If we run the tests, the report informs us that the tests have miserably failed. The probability of collision is terribly high and the distribution is far from being uniform.



Let's improve our hash function by using a simple implementation of Dan Bernstein's famous algorithm.

public override int GetHashCode()
{
    return 33 * value ^ dayOfWeek.GetHashCode();
}

Unsurprisingly, running the tests again makes the report look better :)



Mission accomplished!

Additional information and examples might be found in the Gallio Wiki.

XML Assertions in MbUnit v3

If you have already made some attempts in the past to write unit tests for testing XML output, then you surely noticed that it is not an easy task. Usually, the initial approach is to test the resulting XML data by using a simple text equality assertion. Unfortunately, it does not work very well; mainly because XML contains insignificant whitespaces, comment tags, or self-closing empty elements. Furthermore, we would like sometimes to ignore case of element names, or to ignore the order of the attributes in the same parent element.

For example, are those two fragments equal?

<value x='123' y='456'/>

<VALUE y='456' x='123'></VALUE>

Well, it depends. The name of elements differs by the case, the attributes are in a different order, and one of element is self-closing while the later is not. Nevertheless, it's perfectly reasonable to consider they are equal. And obviously, a unit test which makes use of a text equality assertion will miserably fail because the actual strings are just different.

Fortunately, MbUnit v3.2 proposes some fresh new assertions to test XML data. Basically, the assertions parse the fragments of XML (expected and actual) and compare the resulting trees by taking in account the equality options specified by the user. This is very easy to use. Let's compare our fancy fragments by using Assert.Xml.AreEqual:

[Test]
public void MyXmlTest()
{
   var generator = new MyXmlGenerator();
   string actual = generator.ToXml();
   Assert.Xml.AreEqual("<value x='123' y='456'/>", actual, XmlOptions.Loose);
}

Remark the loose equality options which tells the assertions to ignore comment tags, case of names/values, order of attributes, etc.

More details and examples may be found the Gallio Wiki.

Gallio Wiki

The entire Gallio team has given much focus recently on consolidating the existing documentation around the Gallio ecosystem. In the scope of that collective effort, we have setup a new wiki (http://gallio.org/wiki/).

Although its contents are still a bit sparse at this time, you might find many very interesting articles already. In particular:

• A nice tutorial explaining how to install Gallio and run your firsts unit tests with MbUnit v3 and Icarus.
• A summary of all the release notes of the former versions of Gallio; still invaluable sources of information.
• Some useful references about the built-in tools such as Icarus, Echo, etc.
• And if you want to wander in the Gallio code base, this section about development is a good starting point.

Look forward to reading more...

By the way, feel free to contribute by writing about whatever killer feature you love :)