Niklas Dahlman talks about advanced features of MbUnit v3

It seems that I had totally missed that great presentation by Niklas Dahlman about some advanced features of MbUnit v3. The session was recorded in November 2009 during the Øredev conference.

In particular I like very much the part about the contract verifiers. Niklas shows a couple of very nice examples.

Hash Code Acceptance Contract Verifier in MbUnit v3

Designing a good hash code generation algorithm is a black art. There is no formal and objective way to determine what is the best algorithm for a given scenario. In fact, most of the developers do not know how exactly to implement efficiently hash functions. As you know, hash functions for a given type are implemented in .NET by overriding the method Object.GetHashCode(). Most of us (including myself, I must admit) usually shake up and down the values by shifting and xoring them randomly without knowing exactly if the result will be good enough. It’s even worse: if the implementation is poor, your application will continue to work because a bad hash function does "only" affect performance. So you might even not be able to notice it immediately.

In the other hand, testing a hash function is cumbersome. A good hash function should have the following properties:

• Low probability of collision; meaning that the odds to get two values that produces the same hash code should be as low as possible.
• Hash codes should be distributed uniformly.
• It should achieve "avalanche" by generating hash values wildly different if even a single bit is different in the input key.

Writing unit tests for a hash generator is therefore not a trivial task. Fortunately MbUnit comes to rescue. The soon released v3.2 provides a new contract verifier named Hash Code Acceptance Contract.

The contract verifier adds two child tests to the test fixture. They evaluate the probability of collision and the uniform distribution goodness-of-fit. At this time, the avalanche test is not supported (perhaps in a future release?)

Consider the sample type below. It implements an awful hash function of the additive kind. Let's imagine that according to some imaginary specifications, value is a number between 0 and 9, and dayOfWeek is... well, I just let you find out :)

public class Foo
{
  private readonly int value;
  private readonly DayOfWeek dayOfWeek;

  public Foo(int value, DayOfWeek dayOfWeek)
  {
    this.value = value;
    this.dayOfWeek = dayOfWeek;
  }

  public override int GetHashCode()
  {
    return value + (int)dayOfWeek;
  }
}

Now let's use the contract verifier to evaluate our poor implementation.

[TestFixture]
public class FooTest
{
  [VerifyContract]
  public readonly IContract HashCodeAcceptanceTests = new HashCodeAcceptanceContract<Foo>()
  {
    CollisionProbabilityLimit = CollisionProbability.Low,
    UniformDistributionQuality = UniformDistributionQuality.Good,
    DistinctInstances = DataGenerators.Join(
                          Enumerable.Range(0, 10),
                          Enum.GetValues(typeof(DayOfWeek)).Cast<DayOfWeek>())
                          .Select(o => new Foo(o.First, o.Second))
  };
}

CollisionProbabilityLimit and UniformDistributionQuality are the expected confidence levels. These are probability values between 0 (good) and 1 (bad), but we use here some handy predefined constants to improve readability (more details here and here). You may want to let the default values of 5% if you are not sure.

DistinctInstances must be fed with an enumeration of distinct Foo instances. It's important to understand that the entire evaluation of the hash function is based on the statistical population you decide to provide. Therefore the confidence in the test results will be as good as (or as bad as, for the pessimists) the quality and the representativeness of that population. Two types of scenarios are possible. Either the range of possibilities is finite and reasonably small, or the number of possible distinct instances is nearly infinite or insanely large. In the first case, it's better to provide to the contract verifier all the possible values. Thus you get an exact and complete stochastic evaluation of your hash function. In the second case, you will need to carefully select a representative subset of all the possible values.

In both cases you can feed the contract verifier by using a static method which returns an enumeration.

// ...
DistinctInstances = GetDistinctInstances()
// ...

private static IEnumerable<Foo> GetDistinctInstances()
{
  // ...
}

Or to provide an existing catalog if they can be retrieved from an external data source.

// ...
DistinctInstances = Foo.GetThemAll()
// ...

You may also use the powerful MbUnit data generation framework to easily combine and generate random or sequential values.

// ...
DistinctInstances = DataGenerators.Join(
                      Enumerable.Range(0, 10), 
                      Enum.GetValues(typeof(DayOfWeek)).Cast<DayOfWeek>())
                      .Select(o => new Foo(o.First, o.Second))
// ...

If we run the tests, the report informs us that the tests have miserably failed. The probability of collision is terribly high and the distribution is far from being uniform.



Let's improve our hash function by using a simple implementation of Dan Bernstein's famous algorithm.

public override int GetHashCode()
{
    return 33 * value ^ dayOfWeek.GetHashCode();
}

Unsurprisingly, running the tests again makes the report look better :)



Mission accomplished!

Additional information and examples might be found in the Gallio Wiki.

XML Assertions in MbUnit v3

If you have already made some attempts in the past to write unit tests for testing XML output, then you surely noticed that it is not an easy task. Usually, the initial approach is to test the resulting XML data by using a simple text equality assertion. Unfortunately, it does not work very well; mainly because XML contains insignificant whitespaces, comment tags, or self-closing empty elements. Furthermore, we would like sometimes to ignore case of element names, or to ignore the order of the attributes in the same parent element.

For example, are those two fragments equal?

<value x='123' y='456'/>

<VALUE y='456' x='123'></VALUE>

Well, it depends. The name of elements differs by the case, the attributes are in a different order, and one of element is self-closing while the later is not. Nevertheless, it's perfectly reasonable to consider they are equal. And obviously, a unit test which makes use of a text equality assertion will miserably fail because the actual strings are just different.

Fortunately, MbUnit v3.2 proposes some fresh new assertions to test XML data. Basically, the assertions parse the fragments of XML (expected and actual) and compare the resulting trees by taking in account the equality options specified by the user. This is very easy to use. Let's compare our fancy fragments by using Assert.Xml.AreEqual:

[Test]
public void MyXmlTest()
{
   var generator = new MyXmlGenerator();
   string actual = generator.ToXml();
   Assert.Xml.AreEqual("<value x='123' y='456'/>", actual, XmlOptions.Loose);
}

Remark the loose equality options which tells the assertions to ignore comment tags, case of names/values, order of attributes, etc.

More details and examples may be found the Gallio Wiki.

Gallio Wiki

The entire Gallio team has given much focus recently on consolidating the existing documentation around the Gallio ecosystem. In the scope of that collective effort, we have setup a new wiki (http://gallio.org/wiki/).

Although its contents are still a bit sparse at this time, you might find many very interesting articles already. In particular:

• A nice tutorial explaining how to install Gallio and run your firsts unit tests with MbUnit v3 and Icarus.
• A summary of all the release notes of the former versions of Gallio; still invaluable sources of information.
• Some useful references about the built-in tools such as Icarus, Echo, etc.
• And if you want to wander in the Gallio code base, this section about development is a good starting point.

Look forward to reading more...

By the way, feel free to contribute by writing about whatever killer feature you love :)

Open Source Ad on StackOverflow

An Open Source Ad for Gallio was added recently on StackOverflow. Please upvote for your favorite testing platform here.



Thanks to all in advance!

Gallio test report and TestDriven.Net

If you run usually your tests by using the combination Gallio + TestDriven.Net, then this new feature might interest you. Starting from v3.2.304, it is possible to determine how the test reports are generated by the Gallio test runner for TD.Net.

To achieve that, start the control panel of Gallio; then select the tree node entitled TestDriven.Net » Reports.



For example, you can select the format of the output report (Html, Xml, Text, etc.). Most of the available formats exist in a condensed flavor which omits the passing tests (and thus makes the report shorter and more readable).

An interesting option allows to automatically condense your report when the number of tests is greater than a threshold value. Check the option Auto-Condense Report to enable that functionality.

Once you have run your tests, click on the link at the end of the output log in Visual Studio to open the report.

The Data Generation Framework in MbUnit v3 (Part 3)

MbUnit v3 is able to generate pseudo-random string data for your unit tests. A standard regular expression approach is used to constraint the input.

[TestFixture]
public class LuxembourgMobilePhoneValidatorTest
{
  [Test]
  public void ValidateLuxembourgMobilePhoneNumber(
      [RandomStrings(Count = 20, Pattern = "6[269]1[0-9]{6}")]] string phoneNumber)
  {
    var validator = new LuxembourgMobilePhoneValidator();
    Assert.IsTrue(validator.IsValid(phoneNumber));
  }
}

Dead easy, is'n it?

It's interesting to remark that most often, regular expressions are used to validate a given input. I mean, we first take the input, and then we validate it against a regular expression pattern. But on the contrary, we first have here an existing pattern, and we try to find a random value that matches it. The .NET regular expression framework does not support this scenario. That's why Gallio implements its own very light regular expression parsing engine.

It's light because it does not need to support all those numerous nifty language elements. In fact, we really do not want to support tags and metacharacters that match non-finite sets of elements. For example, how the generator could reasonably handle with the "A+" pattern? Literally, it matches the character 'A' repeated at least once. But what would be the upper limit of the number of times the character is repeated. Would it be reasonable to provide to the test method a string containing a sequence of 1 million A's? Surely not.

That's why only a tiny subset of regular expression metacharacters is supported.

Logical Grouping - Group a part of the expression ((...)).

Explicit Quantifier - Specify the number of times the previous expression must be repeated. Constant ({N}) or Range ({N,M}) syntaxes are both accepted.

Zero Or One Quantifier Metacharacter - 0 or 1 of the previous expression (?). Same effect as {0,1}.

Escape Character - Makes the next character literal instead of a special character (\).