This project is read-only.

Unit Experimentation

Overview

Unit experimentation is the act of writing code to experiment with software in logical units. Experiments can be created to test new ideas when developing or debugging custom software. They can also be created to learn about the behavior and proper usage of existing software.

The primary goal of unit experimentation is understanding. Sometimes it's just for fun, but often it's because you're programming against a particular application programming interface (API) and you need to learn how to use it properly. Experimentation ultimately increases your knowledge of how to use an API to get the desired behavior. Unit experiments can be written against the .NET Framework APIs, third-party APIs and even your own APIs.

A secondary goal of unit experimentation is documentation. When designing your own API you can code unit experiments to illustrate proper usage of common features. Your experiments establish the baseline for others to start writing their own experiments against their particular requirements.

What is a Unit Experiment?

A unit experiment is written in a lab. It can test anything that you want. The software to experiment with can be something that you are developing or it can be existing software that you want to learn. For example, you can code a unit experiment to test an existing API for a hardware device to learn how it works. You can also code an experiment to test your own API for a hardware device to show other developers its proper usage.

What is a Unit?

In design terms, a unit is similar to a usage scenario. When choosing what constitutes the scope of an individual unit, you simply need to consider what part of an API currently interests you, then begin experimenting with it.

In programming terms, a unit can be as simple as a single call to an API or as complicated as a XAML GUI with bindings, timers and web service calls.

The general idea is to keep experiments focused on a single usage scenario. How complex or concise the unit will be depends entirely upon what the scenario is for, but you should try to choose scenarios that are more like building blocks from which larger scenarios can be built; otherwise, you may find that a single unit experiment can become too large and complicated to experiment with any longer.

What is a Lab?

A lab is a container for one or more unit experiments.

The Labs Framework does not require that you indicate which experiments are contained in your labs, so each lab's organization is entirely up to you. The only restriction is that every console lab has a single entry point method named Main. GUI labs have no entry point other than the lab's constructor, but these labs can display custom user interfaces.

The simplest kind of lab consists of a single experiment. A console lab implements an experiment in its Main method, while a GUI lab typically implements its experiment in an event handler, such as the click of a button.

Labs may also contain multiple experiments. GUI labs typically implement multiple experiments in separate event handlers. For example, you could add three buttons: "Experiment 1", "Experiment 2" and "Experiment 3".

Console labs can execute individual experiments in Main. Experiments are typically written as separate instance methods, but you're free to implement and execute them in any way that you choose; e.g., as static methods, methods on different objects, etc. You are not required to indicate the organization of your experiments to the Labs Framework; however, every console lab has an Experiments property that you can optionally override.

The Labs Framework also provides a RunExperiments method that searches for all public, parameterless, void-returning instance and static methods in a console lab and considers them to be individual experiments for that lab. All discovered experiments are returned by the Experiments property. The ExperimentAttribute class is an optional attribute that allows you to set a display name for each method. You can also use System.ComponentModel.DescriptionAttribute to apply descriptions to individual method experiments.

Lab applications targeting WPF, Silverlight and Windows Phone populate a drop-down list with the names of the current lab's experiments and allows users to execute individual experiments in any order. A console lab's Main method is also included as the first item in the list and is treated as a separate experiment that executes the entire lab.

See Method Experiments for more information.

What you do in a console lab's Main method is entirely up to you; however, if you are developing a console lab with multiple experiments, then consider invoking RunExperiments in your Main method and define each experiment as a public, parameterless instance method so that they are discovered automatically by the Labs Framework.

Unit Testing vs. Unit Experimentation

Unit experimentation is out-of-band with respect to the application or library that is under development. This means that the software under development is unaffected by any experiments that you write. Similar to unit tests, all unit experiments are written in a different project that references the projects that are being developed.

Unit experimentation is also out-of-band with respect to unit tests. If you're using Test Driven Development (TDD) or other Agile Programming methodologies to develop your software, then you could still write unit experiments for outside APIs or even for your own APIs whenever you are curious about how they will behave in a realistic environment with unmocked conditions, or to confirm how they should behave, but never to assert their behavior.

The distinction between unit tests and unit experiments can be described as follows:

Assertion vs. Discovery

  • A unit test asserts facts about an API. The developer writing the test knows how the API is supposed to behave, so unit tests simply assert the correct behavior.
  • A unit experiment discovers facts about an API. The developer writing the experiment does not have to know how the API behaves, but instead has assumptions about how it should behave.

Success

  • A successful unit test confirms its assertions.
  • A successful unit experiment confirms the developer's assumptions.

Failure

  • A failed unit test is not a problem with the test, but a problem with the API being tested. The API must be fixed to match the assertions of the test.
  • A failed unit experiment is a problem with the test, because the API being tested is always correct (even when it's incorrect, because buggy software that is not under your control will remain buggy). The experiment can be changed to test different assumptions.
    • Unless an experiment is intended to confirm whether your own API works correctly in a real-world situation, in which case a failed experiment actually does indicate a bug in your API that can be fixed. In this respect your experiment is more like a unit test; however, if you can reproduce the problem in the context of an automated unit test, then it may be a good idea to write a unit test instead of relying on the experiment.

Unit Experimentation APIs

Unit experiments are not designed to make assertions during automated testing, they are designed for you to make observations during manual testing. For this reason the Labs API centers around diagnostic output only, such as tracing and exception handling. There are no APIs similar to unit testing frameworks. Assertions can be made using Debug.Assert. Assumptions can be stated in words by tracing messages to the output console or by including normal code comments.

Other Kinds of Experimentation

Unit experimentation is currently the primary use case for Labs, although you may find Labs useful for other kinds of experimentation as well.

Integration Experimentation

Integration experiments are to integration tests as unit experiments are to unit tests. Combining one or more unit experiments into a single experiment creates an integration experiment.

The terms unit and integration are used loosely here because often labs are actually more like integration experiments than unit experiments. For example, if you experiment against a business object in your application, then you could be inadvertently experimenting against a database object and also your database. This is desirable in many experiments because you want to observe the behavior under controlled circumstances that closely reflect the production environment. Even though the experiment encompasses a path of execution that spans multiple application layers, you can still think of it as a unit of work; e.g., a single function of the business object. An integration experiment might then be one that combines this experiment with another experiment that requires your business object so that it can call its function and bind the result to a UI.

System Experimentation

Prototype applications are system experiments. They contain a subset of features that a real application would have. Each individual feature could have its own unit experiment. The entire system experiment is successful if the prototype functions properly, meaning that all integration experiments and, therefore, all unit experiments also function properly.

Labs Framework is not currently designed for system experimentation, but perhaps it could still be of use.

Last edited Sep 7, 2012 at 10:59 PM by davedev, version 26

Comments

No comments yet.