2008/09/30

Gallio + PowerShell

Nobody is perfect: I am fond of PowerShell. Yes, I know, the PowerShell language syntax is a bit weird sometimes (Do you know any other languages where you use command switches to compare things?), but despite the ugly look of the scripts, this is a very powerful tool.

Gallio has a very handy snap-in for PowerShell, written by Julian Hidalgo and Jeff Brown. It adds a very useful cmdlet which controls the Gallio test runners. The snap-in is configured by the installer. You just need to ensure that the correct install options are checked.



Once Gallio is installed, start the PowerShell console, then load the snap-in in memory with the following command:
Add-PSSnapIn Gallio
Then you can run your test suite by calling Run-Gallio with the appropriate arguments.
Run-Gallio MyTests.dll
More help about the arguments can be found with the following command:
Get-Help Run-Gallio -detailed
One of the most interesting features of PowerShell is that it is entirely object oriented. In that context, the Gallio PowerShell Snap-In provides a complete report object as result. The object is of type Gallio.Runner.TestLauncherResult. Sorry, no online document is available for the moment. But you can get additional details by opening the offline API documentation. There should be a shortcut copied somewhere by the Gallio installer in the start menu. [Update 11/12/2008: Online document is now available on the Gallio website.]



It is easy to look at the result properties, and make the appropriate actions. For instance, you can decide to interrupt the execution of the script if at least one unit test has failed.
Add-PSSnapIn Gallio
$result = Run-Gallio MyTests.dll

if ($result.Statistics.FailedCount -gt 0)
{
Write-Warning "Some unit tests have failed."
Exit
}

2008/09/23

Contract Verifiers - Part IV. - The Comparison Contract Verifier

The comparison contract verifier helps in implementing correctly the generic interface IComparable. It verifies in particular that your type-specific implementation orders instances correctly.
public class Fraction : IComparable<Fration>
{
public Fraction(int numerator, int denominator)
{
// ...
}

// ...
}
As for the other contract verifiers, you must use a dedicated attribute to decorate your test fixture. And as for the equality contract verifier, your test fixture must provide classes of equivalence object instances to be consumed by the contract verifier. The only but important difference here, is that you must feed the method with ordered classes. Each class must contain instances expected to be compared strictly greater than the objects in any previous class. In other words, instances must be put in the ascending order.
[TestFixture]
[VerifyComparisonContract(typeof(Fraction))]
public class FractionTest : IEquivalenceClassProvider<Fration>
{
public EquivalenceClassCollection<Fration> GetEquivalenceClasses()
{
return EquivalenceClassCollection<Fration>.FromDistinctInstances(
new Fraction(1, 6),
new Fraction(1, 5),
new Fraction(1, 4),
new Fraction(1, 3));
};
}
}
The comparison contract verifier checks for the following features:

1. The CompareTo method works as expected. The method must return a negative value if the current object is less than the other object; a positive value if it is greater; or zero if they are equal. Furthermore, if the type is nullable, the contract verifier checks for null reference handling as well. Any non null instance should compares greater than a null reference; and two null references should be equal.
public class Fraction : IComparable<Fration>
{
public int CompareTo(Fraction other)
{
// ...
}
}
Although it is not mandatory, you might decide to override the four comparison operators. The contract verifier is able to check them too. This is an optional feature. You can disable it by setting the named parameter ImplementsOperatorOverload to false.
[TestFixture]
[VerifyComparisonContract(typeof(Fraction), ImplementsOperatorOverload = false)]
public class
FractionTest : IEquivalenceClassProvider<Fration>
{
// ...
}
2. The Greater Than (>) operator overload is correctly implemented and behaves as expected.
public class Fraction : IComparable<Fration>
{
public static bool operator >(Fraction left, Fraction right)
{
// ...
}
}
3. The Greater Than Or Equal (>=) operator overload is correctly implemented and behaves as expected.
public class Fraction : IComparable<Fration>
{
public static bool operator >=(Fraction left, Fraction right)
{
// ...
}
}
4. The Less Than (<) operator overload is correctly implemented and behaves as expected.
public class Fraction : IComparable<Fration>
{
public static bool operator <(Fraction left, Fraction right)
{
// ...
}
}
5. The Less Than Or Equal (<=) operator overload is correctly implemented and behaves as expected.
public class Fraction : IComparable<Fration>
{
public static bool operator <=(Fraction left, Fraction right)
{
// ...
}
}
It is very common to implement equality and comparison in the same time. Should this happen, you can combine several contract verifiers into the same test fixture. This is very handy.
public class Fraction : IComparable<Fration>, IEquality<Fration>
{
}
[TestFixture]
[VerifyComparisonContract(typeof(Fraction))]
[VerifyEqualityContract(typeof(Fraction))]
public class FractionTest : IEquivalenceClassProvider<Fration>
{
// ...
}

2008/09/03

Contract Verifiers - Part III. - The Equality Contract Verifier

The path that leads to a proper implementation of the equality contract has many pitfall traps. Fortunately, the equality contract verifier will help you in avoiding them. For convenience, we will consider as an example, a hypothetical class representing a simple mathematical fraction which implements IEquality<T>.
public class Fraction : IEquality<Fraction>
{
public Fraction(int numerator, int denominator)
{
// ...
}

// ...
}
In order to test the type implementing the generic equality interface, the contract verifier needs some valid type instances. That's why the test fixture must fulfill a two side parts contract.
  • It must have the attribute VerifyEqualityContract.
  • It must implement the interface IEquivalenceClassProvider<T>, which will provide the test runner with various object instances to play with.
using Gallio.Framework;
using MbUnit.Framework;
using MbUnit.Framework.ContractVerifiers;

[TestFixture]
[VerifyEqualityContract(typeof(Fraction))]
public class FractionTest : IEquivalenceClassProvider<Fraction>
{
public EquivalenceClassCollection<Fraction> GetEquivalenceClasses()
{
return EquivalenceClassCollection<Fraction>.FromDistinctInstances(
new Fraction(1, 2),
new Fraction(2, 3),
new Fraction(4, 5));
}
}
The IEquivalenceClassProvider<T> has a single method that you must use to return a collection of equivalence classes for the evaluated type. An equivalence class is a collection of object instances that are expected to be equal (The notion of equality is of course to be understood in the sense of your own equality contract). In the simplest scenarios, it is generally sufficient to provide several distinct object instances to be compared together. Use the factory method FromDistinctInstances to construct such a collection of equivalence classes. In the previous example, we provide 3 equivalence classes, each containing 1 single object.

However, in more complex scenarios, we want to verify that different object instances "representing" the same value are actually equal. For instance, the fraction 1/2 should be equal to the fraction 2/4 or 3/6. Instead of using the factory method, use the constructor to provide some equivalence classes containing a variable number of objects.
[TestFixture]
[VerifyEqualityContract(typeof(Fraction))]
public class FractionTest : IEquivalenceClassProvider<Fraction>
{
public EquivalenceClassCollection<Fraction> GetEquivalenceClasses()
{
return new EquivalenceClassCollection<Fraction>(
new EquivalenceClass<Fraction>(
new Fraction(1, 2),
new Fraction(2, 4),
new Fraction(3, 6)),
new EquivalenceClass<Fraction>(
new Fraction(2, 3),
new Fraction(4, 6)),
new EquivalenceClass<Fraction>(
new Fraction(4, 5)))
}
}
Now that we have defined our contract verifier, let's see what it will check for.

1. It verifies that the method GetHashCode returns the same value for equal objects. You should override the method and return an identical value for all the objects located in the same equivalence class. However, the contrary is not true: it is not required that the resulting hash value for unequal objects be different (although it would be a good idea, for efficiency reasons)
public class Fraction : IEquality<Fraction>
{
public override int GetHashCode()
{
// ...
}
}
2. It verifies that the method Object.Equals(object) behaves as expected. The method should return true when two instances of the same equivalence class are compared together, or when an instance is compared to itself. For nullable types (reference types, or value types nested in a Nullable class), it verifies you dealt correctly with null references. The method should return False when called against a null reference.
public class Fraction : IEquality<Fraction>
{
public override bool Equals(object obj)
{
// ...
}
}
3. Same verifications as point 2; but for the strongly typed version of the Equals method (which is actually the only method that the IEquatable<T> interface requires you to implement).
public class Fraction : IEquality<Fraction>
{
public bool Equals(Fraction other)
{
// ...
}
}
Whenever implementing IEquatable<T>, it is usually recommended to overload the two equality operators as well. By default, the equality contract verifier will evaluate them too. If your class does not override those operators, consider disabling that feature by setting to false the named parameter ImplementsOperatorOverload.
[TestFixture]
[VerifyEqualityContract(typeof(Fraction), ImplementsOperatorOverload = false)]
public class FractionTest : IEquivalenceClassProvider<Fraction>
{
// ...
}
4. It verifies that the equality operator (==) behaves as expected. The verifications are the same as for points 2 and 3. For nullable types, it is also verified that two null references are considered to be equal.
public class Fraction : IEquality<Fraction>
{
public static bool operator ==(Fraction left, Fraction right)
{
// ...
}
}
5. It verifies that the inequality operator (!=) behaves as expected.
public class Fraction : IEquality<Fraction>
{
public static bool operator !=(Fraction left, Fraction right)
{
// ...
}
}

2008/08/25

Contract Verifiers - Part II. - The Exception Contract Verifier

Designing custom exceptions is not a so easy task. The recommended guidelines are relatively strict; and writing unit tests for them is fastidious. That is why the exception contract verifier is useful. It helps testing your custom exceptions, and catching potential design and behavioral problems.

The contract verifier is available as a companion attribute for your test fixture. It adds the necessary test methods, which verify that your exception is well designed and behaves as expected.
public class MyCustomException : Exception
{
// ...
}
using Gallio.Framework;
using MbUnit.Framework;
using MbUnit.Framework.ContractVerifiers;

[TestFixture]
[VerifyExceptionContract(typeof(MyCustomException))]
public class MyCustomExceptionTest
{
}
Although the test fixture contains no explicit test methods, the test runner will execute 5 tests provided by the contract verifier attribute. Each of them corresponds to a specific guideline:

1. Provides a public default parameter-less constructor.
2. Provides a public constructor taking one single parameter for the error message .
3. Provides a public constructor taking two parameters for the error message and an inner exception.

public class MyCustomException : Exception
{
public MyCustomException()
{
// ...
}

public MyCustomException(string message)
{
// ...
}

public MyCustomException(string message, Exception innerException)
{
// ...
}
}
If for some reason, you decide that your custom exception should handle with construction differently, you can disable the 3 tests above by setting to false the named parameter ImplementsStandardConstructors.
[TestFixture]
[VerifyExceptionContract(typeof(MyCustomException), ImplementsStandardConstructors = false)]
public class MyCustomExceptionTest
{
}
4. Decorates your custom exception with the attribute System.SerializableAttribute.
[Serializable]
public class MyCustomException : Exception
{
}
5. Implements a protected constructor for data serialization.
public class MyCustomException : Exception, ISerializable
{
protected MyCustomException(SerializationInfo info, StreamingContext context)
{
// ...
}
}
Again, in some scenarios, you might decide to not support the serialization process (if your code base targets the compact framework, for instance). Disable the serialization tests by setting to false the named parameter ImplementsSerialization.
[TestFixture]
[VerifyExceptionContract(typeof(MyCustomException), ImplementsSerialization = false)]
public class MyCustomExceptionTest
{
}
When the serialization tests are enabled, the test runner will verify the preservation of data during a complete round trip serialization operation. This will be done for all the constructors; including the 3 recommended standard constructors described above.

The following sample shows the complete code for a general-purpose custom exception; and the test fixture verifying it.
[Serializable]

public class MyCustomException : Exception, ISerializable
{
public MyCustomException()
{
}

public MyCustomException(string message)
: base(message)
{
}

public MyCustomException(string message, Exception innerException)
: base(message, innerException)
{
}

protected MyCustomException(SerializationInfo info, StreamingContext context)
: base(info, context)
{
}

public override void GetObjectData(SerializationInfo info, StreamingContext context)
{
base.GetObjectData(info, context);
}
}
[TestFixture]
[VerifyExceptionContract(typeof(MyCustomException))]
public class MyCustomExceptionTest
{
}
And here is the report generated by Gallio for that test fixture:

2008/08/19

Contract Verifiers - Part I. - Introduction

Writing unit tests is costly. Surely less costly than not writing them, but costly too. A commonly accepted trade-off consists in not testing trivial features, or worst, repeating features. You know, all those dozens of little classes that implement repeatedly a roughly similar stuff: custom exceptions, event arguments, equatable, comparable, or disposable interfaces, etc. If you write unit tests, you know how painful and uninteresting it is to test those objects.

Honestly, look at the last project you have worked on. Do you see all those custom exceptions you have written? Have you really tested their serialization and unserialization support? Look at these classes implementing the generic IEquatable interface. Have you verified that the equality behaves correctly in all the possible ways (to a null reference, to itself, commutability of the operation, etc?) If the answer is no, then I sympathize. If yes, well, it was a so boring job, wasn't it?



The next release of Gallio (alpha 4) will come with a cool feature called Contract Verifiers. Actually, it is already available in the alpha 3, but under a preliminary form which will be probably marked as obsolete in the future (or perhaps simply removed).
[Update; Actually, Jeff has just removed it from the repository.]

The principle of the contract verifiers is simple. Basically, a contract verifier is a set of predetermined test methods that you can attach to a given test fixture without actually writing them (we could describe it as a kind of Mixin). The test methods will verify for you that the object fulfills the given implementation contract. It is an all-in-one functionality because the test methods have already been written for you. They also support some configuration; so they can be used in different scenarios.

Let's take an example. Imagine you need to write a class A implementing a well-known interface, or deriving from a well-known base class. Let's call it X. You will have to override the necessary members of X, so that A works correctly; and to write a bunch of unit tests that will verify that the functionalities of X (let's call them x1, x2 and x3) work as expected in your implementation of A. Fortunately, Gallio comes with a contract verifier for X. We will name it XContractVerifier. Unit testing A comes suddenly easy:

[TestFixture]
[TestsOn(typeof(A)]
[XContractVerifier(option1=blabla, option2=blabla)]
public class ATest
{
// My other own tests here...
}

Thanks to the XContractVerifier attribute, and during the execution of the test fixture, the test runner of Gallio will add behind the scene, the necessary test methods that will verify that A implements correctly x1, x2, and x3.

Gallio Alpha 4 includes three built-in contract verifiers (it is a beginning):
It is interesting to notice that the internal working of the contract verifiers is based on the remarkable capability of the Gallio exploration engine to dynamically attach child test methods to a given fixture; although those test methods do not physically exist.

2008/07/25

Contributing to Gallio

It is not the first time I have a close look at the source code of an open source project. My hard drive has a dedicated folder where I can find the internals of NHibernate, Paint.NET, Rhino, or Castle. Studying and analyzing good code, and understanding the whys and hows of its internal design, is worth the time you will invest. There is so much to learn from the guru coders who lead those projects.

However, it is the first time I have the desire to contribute significantly to such a project (Other than submitting small patches which fix minor issues). I discovered the Gallio Automation Platform early in its alpha 1 stage. It was an exciting project with tons of innovative concepts; and I have eagerly followed the development phases since that very moment.

When Jeff Brown, the lead developer of Gallio, mentioned in his blog the existence of the contract verifiers, I immediately thought that it was a cool feature that I would like to work on. Two months later and with Jeff's invaluable help, I am in the process of refactoring completely the implementation of the contract verifiers for MbUnit v3, based on Jeff's own road map and ideas. I am more than happy to work on such a great project during my free time.

Expect to learn about contract verifiers later on this blog.

2008/07/05

Every interface is public

A few days ago, I posted a comment on timm's blog. I made a remark about the difficulties that most .NET developers had and still have when they have to handle with Stream objects to read and write text files; especially in the early days of the .NET framework, when methods such as File.ReadAllText and File.ReadAllLines did not exist yet.

Then, I remembered reading that on the book written by Brad Adams and Krzysztof Cwalina: "Framework Design Guidelines".



Although published already two years ago, this is still an excellent reference about naming conventions, sane type design and class library extensibility. Perhaps it would deserve a second edition with additional stuff about C#3, Linq, WPF, WCF, etc. This book is also a source of valuable comments and amusing little stories from various actors in the .NET framework world, explaining why some classes were designed the way they are, or which were the difficulties they met while designing the class library. Delightful…

However, there is a little point in that book which I tend to disagree with. It is apparently only a detail, but which hides perhaps an entirely different philosophical point of view about code design. The authors say about naming guidelines:
Although adopting these naming conventions as general code development guidelines would result in more consistent naming throughout your code, you are required only to apply them to APIs that are publicly exposed.

As a corporate developer, I write code that has little chance to be read or to be reused by other developers outside my work place. The question I ask myself, is what exactly "publicly exposed APIs" mean? Technically speaking, it should be any type or member with a public scope. But actually, when my colleague reads my code, he has also access to objects with private or internal scope. When I download the code of an open-source project such as Rhino or the promising Gallio, I also have access to internal types. In VS2008, we have also the possibility to see the source code of the class library, which has not a public scope. And finally, when I stop working for a while on a project of mine, the person I am when I resume the job is not exactly the same. I have perhaps forgotten how the system was designed; so I come with a new fresh eye and new recently acquired experience.

I believe that "publicly exposed APIs" is not synonym of "types and members with a public scope". In fact, I tend to consider every single line of code as public at a certain level: public to other anonymous developers, public to your colleagues, or only public to you. As soon as a line of code is written, it can obviously be read as well. Maybe not by everyone, but at least by you and the other developers who work with you.

That's why I think it does not make sense to apply partially such a guideline. It's like pushing dust under the carpet. For sure, people passing next to the house will never know you do that. They will only see the "public interface", that is the front of your home. But your family might discover it, or your neighbor, who will visit you one day or another. But more important, YOU will know it. If you can live with that mess in your code, I have the bad feeling that you are not rigorous enough to become a good developer.