Announcing Gallio and MbUnit v3.2.1

We are pleased to announce that a maintenance release of Gallio/MbUnit is now available. This release mainly contains many bug fixes and little enhancements. But it also features the long-awaited support to Jetbrains dotCover. See the release notes for more details.

Please visit the Gallio website Downloads page to get the binaries, or grab them directly from here:


MbUnit v3 to support (N)Hamcrest

Recently, Graham Hay did port the well-known Hamcrest library to .NET. The new version of that library was judiciously ;) named NHamcrest. It is also fully integrated to Gallio/MbUnit with Assert.That.

Try it out by downloading the latest v3.3 daily build and give us feedback and suggestions.
public void NHamcrestExample()
var array = new[] {"red", "green", "blue"};
Assert.That(array, Is.InstanceOf(typeof(string[])));
Assert.That(array, Has.Items(new[] { "green", "red" }));
Assert.That(array, Has.Item<string>(Starts.With("bl")));


Testing Custom Data Source Attributes in MbUnit v3.2

In a recent post, I explained how to create a custom data source attribute for MbUnit. But before using it, it's certainly safer to test it. The method described below is the same as the one which is applied in the MbUnit test project itself. It is widely used to verify that the built-in attributes of MbUnit behave as expected. The principle is the following:
  1. Create a nested explicit sample test fixture which consumes the attribute under test. It must be marked as explicit, so that it will not be taken in account by the primary test runner.
  2. Create a regular unit test which launches an inner isolated test runner, and runs the sample fixture.
  3. Retrieve the output of the inner test runner and assert over the test log.
The Gallio framework has everything you need to create an isolated test runner. But in order to make the things easier, the Gallio SDK (you can find the SDK under %gallio_install_path%\sdk) contains a couple of handy helper classes for that very purpose. It provides in particular a BaseTestWithSampleRunner class that you can use as a base class of you main fixture, and a [RunSample] attribute to easily target the nested explicit sample fixtures.

Here is a simple example that shows how to test the [BooleanData] attribute that we did create last time.
[TestFixture, RunSample(typeof(SampleFixture))]
public class BooleanDataAttributeTest : BaseTestWithSampleRunner
public void Test()
var runs = GetTestStepRuns(typeof(SampleFixture), "Test");
var logs = runs.Select(GetLog).Where(x => x.Length > 0);
Assert.AreElementsEqualIgnoringOrder(new[] { "value=True", "value=False" }, logs);

[TestFixture, Explicit]
internal class SampleFixture
public void Test([BooleanData] bool value)
TestLog.Write("value={0}", value);
Want to know more? Be sure to read this page in the Gallio wiki.


Announcing Gallio and MbUnit v3.2

We are pleased to announce the release of Gallio & MbUnit v3.2. Please visit the Gallio website download page. This is a major release with many new features and enhancements. See the release notes for more details.

Be sure to read Andy's blog post too.


Writing Custom Data Source Attributes in MbUnit v3.2

The extensibility of the Gallio platform is simply amazing. You can virtually extend any part of the system. It goes from a simple plugin that provides new handy functionalities to a full-blown adapter for your fancy testing framework. Today, I would like to explain how easy it is to extend the MbUnit framework with a custom data source attribute.

MbUnit has many useful built-in data source attributes which might be used to create powerful data-driven tests. The most popular attributes are certainly [Row] and [Column].

Now let's imagine you get bored with writing that kind of tests:
public void Mytest(
[Column(true, false)] bool flag1,
[Column(true, false)] bool flag2,
[Column(true, false)] bool flag3)
// ...
Imagine how beautiful would be the world if you could write the following instead:
public void Mytest(
[BooleanData] bool flag1,
[BooleanData] bool flag2,
[BooleanData] bool flag3)
// ...
Unfortunately, this data source attribute does not exist in MbUnit.

So let's make it happen!

Creating a custom data source is very easy. Basically you simply need to derive from MbUnit.Framework.DataAttribute, and to override the virtual method PopulateDataSource.
[AttributeUsage(PatternAttributeTargets.DataContext, AllowMultiple = false, Inherited = true)]
public class BooleanDataAttribute : DataAttribute
protected override void PopulateDataSource(IPatternScope scope, DataSource dataSource, ICodeElementInfo codeElement)
dataSource.AddDataSet(new ValueSequenceDataSet(new object[] { true, false }, GetMetadata(), false));
That's all. Compile and run your test happily!

You might find more inspiration by examining the actual implementation of existing built-in attributes such as [EnumData] or [RandomStrings] which is slightly more complicated as it relies on the underlying Gallio data generation framework.

Next time, I will explain how to properly test you custom data source attribute by using the Gallio SDK.


Data-Driven Testing in MbUnit v3

MbUnit was mostly known in the past as the testing framework dedicated to data-driven testing. This is certainly due to the very popular [Row] attribute. But unsurprisingly this statement is still true in version 3 as the support for data-driven testing is still certainly one the most powerful feature of Gallio/MbUnit v3.

But the famous [Row] attribute is not the only way to create data-driven tests. We have written several articles and tutorials in the Gallio Wiki which cover that wide topic. All the pages have not been written yet, but you will find many useful samples and details already. They explain how to use the [Row], [Column] and [Factory] attributes, how to combine them and how to flavor them with tasty parameters. In the next coming weeks, you will read more about external and generative data sources as well.


Assert.Count in MbUnit v3.2

Assert.Count is a last-minute addition made to Gallio/MbUnit v3.2 just before passing to RC. Be sure to read the nice article that Vadim Kreynin has written about that new convenient assertion.


Gallio & MbUnit 3.2 RC

Graham has unleashed today Gallio v3.2 RC in the wild:
The latest build of Gallio & MbUnit 3.2 (build 517) is considered a Release Candidate, you can find it here. Release notes are on the wiki. This means no new features will be added, but obviously any critical bugs will be fixed before the official release.

We're aware that it's been a long time since the last release, and we're hoping to push more frequent smaller releases over the coming months.

Please download it and let us know what you think!


Gallio v3.2 + ReSharper 5.1

As you may know, Graham Hay is one of the major contributors of the Gallio project. He has committed recently an amazing amount of work to make Gallio support the latest version of JetBrains ReSharper.

The version 5.1 of R# has not been officially released yet; but you can run your tests with Gallio and that very popular Visual Studio extension already.

Download here the latest build of the Gallio bundle.


Niklas Dahlman talks about advanced features of MbUnit v3

It seems that I had totally missed that great presentation by Niklas Dahlman about some advanced features of MbUnit v3. The session was recorded in November 2009 during the Øredev conference.

In particular I like very much the part about the contract verifiers. Niklas shows a couple of very nice examples.


Hash Code Acceptance Contract Verifier in MbUnit v3

Designing a good hash code generation algorithm is a black art. There is no formal and objective way to determine what is the best algorithm for a given scenario. In fact, most of the developers do not know how exactly to implement efficiently hash functions. As you know, hash functions for a given type are implemented in .NET by overriding the method Object.GetHashCode(). Most of us (including myself, I must admit) usually shake up and down the values by shifting and xoring them randomly without knowing exactly if the result will be good enough. It’s even worse: if the implementation is poor, your application will continue to work because a bad hash function does "only" affect performance. So you might even not be able to notice it immediately.

In the other hand, testing a hash function is cumbersome. A good hash function should have the following properties:
  • Low probability of collision; meaning that the odds to get two values that produces the same hash code should be as low as possible.
  • Hash codes should be distributed uniformly.
  • It should achieve "avalanche" by generating hash values wildly different if even a single bit is different in the input key.
Writing unit tests for a hash generator is therefore not a trivial task. Fortunately MbUnit comes to rescue. The soon released v3.2 provides a new contract verifier named Hash Code Acceptance Contract.

The contract verifier adds two child tests to the test fixture. They evaluate the probability of collision and the uniform distribution goodness-of-fit. At this time, the avalanche test is not supported (perhaps in a future release?)

Consider the sample type below. It implements an awful hash function of the additive kind. Let's imagine that according to some imaginary specifications, value is a number between 0 and 9, and dayOfWeek is... well, I just let you find out :)
public class Foo
private readonly int value;
private readonly DayOfWeek dayOfWeek;

public Foo(int value, DayOfWeek dayOfWeek)
this.value = value;
this.dayOfWeek = dayOfWeek;

public override int GetHashCode()
return value + (int)dayOfWeek;
Now let's use the contract verifier to evaluate our poor implementation.
public class FooTest
public readonly IContract HashCodeAcceptanceTests = new HashCodeAcceptanceContract<Foo>()
CollisionProbabilityLimit = CollisionProbability.Low,
UniformDistributionQuality = UniformDistributionQuality.Good,
DistinctInstances = DataGenerators.Join(
Enumerable.Range(0, 10),
.Select(o => new Foo(o.First, o.Second))
CollisionProbabilityLimit and UniformDistributionQuality are the expected confidence levels. These are probability values between 0 (good) and 1 (bad), but we use here some handy predefined constants to improve readability (more details here and here). You may want to let the default values of 5% if you are not sure.

DistinctInstances must be fed with an enumeration of distinct Foo instances. It's important to understand that the entire evaluation of the hash function is based on the statistical population you decide to provide. Therefore the confidence in the test results will be as good as (or as bad as, for the pessimists) the quality and the representativeness of that population. Two types of scenarios are possible. Either the range of possibilities is finite and reasonably small, or the number of possible distinct instances is nearly infinite or insanely large. In the first case, it's better to provide to the contract verifier all the possible values. Thus you get an exact and complete stochastic evaluation of your hash function. In the second case, you will need to carefully select a representative subset of all the possible values.

In both cases you can feed the contract verifier by using a static method which returns an enumeration.
// ...
DistinctInstances = GetDistinctInstances()
// ...

private static IEnumerable<Foo> GetDistinctInstances()
// ...
Or to provide an existing catalog if they can be retrieved from an external data source.
// ...
DistinctInstances = Foo.GetThemAll()
// ...
You may also use the powerful MbUnit data generation framework to easily combine and generate random or sequential values.
// ...
DistinctInstances = DataGenerators.Join(
Enumerable.Range(0, 10),
.Select(o => new Foo(o.First, o.Second))
// ...
If we run the tests, the report informs us that the tests have miserably failed. The probability of collision is terribly high and the distribution is far from being uniform.

Let's improve our hash function by using a simple implementation of Dan Bernstein's famous algorithm.
public override int GetHashCode()
return 33 * value ^ dayOfWeek.GetHashCode();
Unsurprisingly, running the tests again makes the report look better :)

Mission accomplished!

Additional information and examples might be found in the Gallio Wiki.


XML Assertions in MbUnit v3

If you have already made some attempts in the past to write unit tests for testing XML output, then you surely noticed that it is not an easy task. Usually, the initial approach is to test the resulting XML data by using a simple text equality assertion. Unfortunately, it does not work very well; mainly because XML contains insignificant whitespaces, comment tags, or self-closing empty elements. Furthermore, we would like sometimes to ignore case of element names, or to ignore the order of the attributes in the same parent element.

For example, are those two fragments equal?
<value x='123' y='456'/>
<VALUE y='456' x='123'></VALUE>
Well, it depends. The name of elements differs by the case, the attributes are in a different order, and one of element is self-closing while the later is not. Nevertheless, it's perfectly reasonable to consider they are equal. And obviously, a unit test which makes use of a text equality assertion will miserably fail because the actual strings are just different.

Fortunately, MbUnit v3.2 proposes some fresh new assertions to test XML data. Basically, the assertions parse the fragments of XML (expected and actual) and compare the resulting trees by taking in account the equality options specified by the user. This is very easy to use. Let's compare our fancy fragments by using Assert.Xml.AreEqual:
public void MyXmlTest()
var generator = new MyXmlGenerator();
string actual = generator.ToXml();
Assert.Xml.AreEqual("<value x='123' y='456'/>", actual, XmlOptions.Loose);
Remark the loose equality options which tells the assertions to ignore comment tags, case of names/values, order of attributes, etc.

More details and examples may be found the Gallio Wiki.


Gallio Wiki

The entire Gallio team has given much focus recently on consolidating the existing documentation around the Gallio ecosystem. In the scope of that collective effort, we have setup a new wiki (http://gallio.org/wiki/).

Although its contents are still a bit sparse at this time, you might find many very interesting articles already. In particular:
Look forward to reading more...

By the way, feel free to contribute by writing about whatever killer feature you love :)


Open Source Ad on StackOverflow

An Open Source Ad for Gallio was added recently on StackOverflow. Please upvote for your favorite testing platform here.

Thanks to all in advance!


Gallio test report and TestDriven.Net

If you run usually your tests by using the combination Gallio + TestDriven.Net, then this new feature might interest you. Starting from v3.2.304, it is possible to determine how the test reports are generated by the Gallio test runner for TD.Net.

To achieve that, start the control panel of Gallio; then select the tree node entitled TestDriven.Net » Reports.

For example, you can select the format of the output report (Html, Xml, Text, etc.). Most of the available formats exist in a condensed flavor which omits the passing tests (and thus makes the report shorter and more readable).

An interesting option allows to automatically condense your report when the number of tests is greater than a threshold value. Check the option Auto-Condense Report to enable that functionality.

Once you have run your tests, click on the link at the end of the output log in Visual Studio to open the report.


The Data Generation Framework in MbUnit v3 (Part 3)

MbUnit v3 is able to generate pseudo-random string data for your unit tests. A standard regular expression approach is used to constraint the input.
public class LuxembourgMobilePhoneValidatorTest
public void ValidateLuxembourgMobilePhoneNumber(
[RandomStrings(Count = 20, Pattern = "6[269]1[0-9]{6}")]] string phoneNumber)
var validator = new LuxembourgMobilePhoneValidator();
Dead easy, is'n it?

It's interesting to remark that most often, regular expressions are used to validate a given input. I mean, we first take the input, and then we validate it against a regular expression pattern. But on the contrary, we first have here an existing pattern, and we try to find a random value that matches it. The .NET regular expression framework does not support this scenario. That's why Gallio implements its own very light regular expression parsing engine.

It's light because it does not need to support all those numerous nifty language elements. In fact, we really do not want to support tags and metacharacters that match non-finite sets of elements. For example, how the generator could reasonably handle with the "A+" pattern? Literally, it matches the character 'A' repeated at least once. But what would be the upper limit of the number of times the character is repeated. Would it be reasonable to provide to the test method a string containing a sequence of 1 million A's? Surely not.

That's why only a tiny subset of regular expression metacharacters is supported.

  • Logical Grouping - Group a part of the expression ((...)).
  • Explicit Quantifier - Specify the number of times the previous expression must be repeated. Constant ({N}) or Range ({N,M}) syntaxes are both accepted.
  • Zero Or One Quantifier Metacharacter - 0 or 1 of the previous expression (?). Same effect as {0,1}.
  • Escape Character - Makes the next character literal instead of a special character (\).