Massive-scale Online Software Development

Cutting right to the chase: Would it be possible to create software, entirely developed and moderated by an open community?

Call it democratic software development, or open source on steroids if you will. While discussing this the default answer I usually get is “it can’t be done”, which is why I gladly filed this post under the newly created category “Crazy Ideas”. Nevertheless, I find it a valuable exercise to discuss any nutcase ideas, in order to evaluate how far-fetched they actually are.

A person with a new idea is a crank until the idea succeeds. – Mark Twain

So what would such a system look like? What would be some of the requirements?

Easily accessible.
And when I say easy, I mean it. You shouldn’t have to download the repository first. You shouldn’t have to set up a development environment. It should run as a web service on the cloud (more buzzwords coming up!). A user account and an internet connection is all you need to get going.

Motivate people to participate.
Ever heard of gamification? “Gamification is the use of game design techniques and mechanics to solve problems and engage audiences.” If you are a software developer, chances are you ended up on Stack Overflow at some point. It’s a Q&A site for programmers which is quickly becoming one of the main resources for help for professional programmers. Stack Overflow incorporates many aspects of gamification, and it’s mere existence shows the power of it. A significant amount of developers is prepared to share and learn in this fun environment. Quality content is pushed to the top via a voting system, while erroneous posts are addressed by the community.

Divide work in small enough tasks.
The key to dividing work across many people is to divide it in such a way that any person only has to implement one small aspect of it at a time. Traditional software development where somebody develops a feature from a to z won’t work. One programming paradigm which at first sight seems extremely suitable for this is functional programming. A person could implement functions, and define new functions on which he relies. Combining this with aspects from test driven development where the caller has to comment and write tests for the desired function would result in automated testing.

Without worrying about the specifics too much (it’s just a nutcase idea after all) consider what the possibilities would be. In Luis von Ahn’s great TED talk the CAPTCHA inventor discusses how he re-purposed CAPTCHA in order to digitize books. Around 2,5 million books a year can be digitized through this massive-scale collaborative effort. Their next project indicates this doesn’t have to be limited to really mundane tasks. They are now working on translating the web!

Moderation guided by conventions.
Conventions are important in a group effort. Unfortunately, when discussing programming conventions people most often discuss naming and formatting conventions, while there are plenty of other important conventions to agree on. This will most likely be the topic of one of my future posts. Conventions should be as unambiguous as possible in order to know where to expect a certain piece of code, or where to place it. Conventions like these could be agreed upon through a democratic process, which seems to be working pretty well for Stack Overflow through its meta site. This allows for community moderation following the guidelines established by the community.

Couple all the separate work together into one entity.
Going from a set of loosely coupled functions to a working library would result in plenty of extra challenges, but also opportunities. Since nobody wants an all encompassing library just to use part of its functionality, the system should allow you to extract just the functionality you are interested in.

Beyond the idea

Well, … I went a bit further and attempted to start a small proof of concept. I figured the Stack Exchange platform on which Stack Overflow runs already encompasses much of the desired functionality, and creating a small scale library on it would be possible.  The idea was to create an extension method library for C#, which exists primarily out of a set of functions. Requesting new Stack Exchange sites is possible through Area 51. Not unexpectedly, my idea got shot down since it doesn’t fit the intended Q&A format. Oh well, … one can only try.

UPDATE:

At the UIST 2014 conference, Thomas D. LaToza presented “Microtask Programming: Building Software with a Crowd”a system encompassing many of these ideas and actually evaluating them, resulting in usable code. The paper is available on ACM.

The code formatting fallacy

Time to stir my category cloud on the right a bit, and publish a post under a category I’ve so far only used once. Hopefully it will become clear what Language Oriented Programming (LOP) has got to do with code formatting after this post.

There are several holy wars on code formatting:

… and  many more.

All these wars are about how text should be formatted. The very notion of code being text is so ingrained in programmers, that they often can’t think outside the box. What if the actual problem is code being text?

The problem is that text editors are stupid and don’t know how to work with the underlying graph structure of programs. But with the right tools, the editor could work directly with the graph structure, and give us freedom to use any visual representation we like in the editor. – Sergey Dmitriev, cofounder and CEO of JetBrains Inc.

This is just one of many interesting statements in the must-read article by Sergey introducing LOP. Once you consider the possibility of separating code and it’s representation, most of the code formatting discussions become obsolete.

  • Identifiers can contain spaces, or can be represented differently based on preferences. Heck, they could be icons in a diagram.
  • Visually separating a piece of code for readability would be possible without having to split into smaller functions.
  • Multiple exit points of a function can be visualized in a diagram.

… and you can think of many more.

Modern IDEs like Visual Studio are slowly adopting alternate visualizations for code made possible by using extensions. As useful as some of these extensions are, they inherit the limitations of having to work on top of text.

If the progress on JetBrain’s Meta Programming System (MPS) is any indication of how feasible LOP will be in the future, my guess is we will be hearing a lot more from this paradigm in the years to come. A new actionscript editor, Realaxy, seems to be more than capable of competing with existing editors, and is built entirely on top of MPS.

UPDATE: Markus Voelter announced mbeddr on his blog, the C language made extensible thanks to MPS.

UPDATE: It seems the Realaxy website no longer exists (I don’t know why), thus I linked to the wikipedia article instead.

Like me you might not have time to try out MPS properly, but I can highly recommend subscribing to their blog. New screencasts are appearing at a regular basis, showcasing features ranging from implementing new language keywords to creating a visual editor for a state machine.

Attribute metabehavior

Attributes in .NET can be used to add metadata to language elements. In combination with reflection they can be put to use in plenty of powerful scenarios. A separate parser can process the data added and act upon the annotations in any way desired. A common example of this is a serializer which knows how to serialize an instance of a class based on attributes applied to its members. E.g. a NonSerialized attribute determines the member doesn’t need to be serialized.

In my previous post I promised I would show some interesting scenarios where the mandatory enum values in my solution, needed to identify different dependency properties (DPs), can be put to good use. It is not required to grasp the entire subject previously discussed, but it does help to know about DPs and their capabilities. Simply put, they are special properties (used by XAML) which can notify whenever they are changed, allowing to bind other data to them. They can also be validated (check whether a value assigned to them is valid or not) and coerced (adjusting the value so it satisfies a certain condition). I consider these capabilities to be metadata, describing particular behavior associated with a given property. After extensive tinkering I found a way to express this behavior in an attribute, but it wasn’t easy. Ordinarily WPF requires you to pass callback methods along when ‘registering’ the DPs, which implement the desired behavior.

Two solutions can be considered:

  1. The parser knows how to interpret the metadata and acts accordingly.
  2. The attribute implements the actual behavior itself, and is simply called by the parser.

Solution 1 is the easiest to implement, and is how attributes are used most often. Solution 2 however has a great added benefit. It allows you to apply the strategy pattern. Unlike solution 1, the parser doesn’t need to know about the specific implementation, but only the interface. Additional behaviors can be implemented and used without having to modify the parser. In contrast to simple metadata, an actual behavior is attached, hence I am dubbing this metabehavior. I will discuss the loopholes you have to jump through to achieve this in a next post. For now, consider the following examples to see how it could be used.

A regular expression validation of a dependency property can be used as follows:

[DependencyProperty( Property.RegexValidation, DefaultValue = "test" )]
[RegexValidation( "test" )]
public string RegexValidation { get; set; }

Coercing a value within a certain range, defined by two other properties can be used as follows:

[DependencyProperty( Property.StartRange, DefaultValue = 0 )]
public int StartRange { get; set; }

[DependencyProperty( Property.EndRange, DefaultValue = 100 )]
public int EndRange { get; set; }

[DependencyProperty( Property.CurrentValue )]
[RangeCoercion( typeof( int ), Property.StartRange, Property.EndRange )]
public int CurrentValue { get; set; }

The greatest thing about all this, is that new behaviors can be implemented by extending from ValidationHandlerAttribute and CoercionHandlerAttribute respectively, albeit with some added complexities due to the limitations of attributes which I will discuss later.

Casting to less generic types

… because yes, there are valid use cases for it! Finally I found the time to write some unit tests, followed by fixing the remaining bugs, and am now excited to report on the result which effectively allows you to break type safety for generic interfaces, if you so please. Previously I discussed the variance limitations for generics, and how to create delegates which overcome those limitations. Along the same lines I will now demonstrate how to overcome those limitations for entire generic interfaces.

Consider the following interface which is used to check whether a certain value is valid or not.

public interface IValidation<in T>
{
    bool IsValid( T value );
}

Notice how T is made contravariant by using the in keyword? Not only values of type T can be validated, but also any extended type. This recent feature of C# won’t help a bit in the following scenario however. During reflection you can only use this interface when you know the complete type, including the generic type parameters. In order to support any type, you would have to check for any possible type and cast to the correct corresponding interface.

object validator;  // An object known to implement IValidation
object toValidate; // The object which can be validated by using the validator.

if ( validator is IValidation<string> )
{
    IValidation<string> validation = (IValidation<string>)validator;
    validation.IsValid( (string)toValidate );
}
else if ( validator is IValidation<int> )
{
    IValidation<int> validation = (IValidation<int>)validator;
    validation.IsValid( (int)toValidate );
}
else if ...

Hardly entertaining, nor maintainable. What we actually need is covariance in T, or at least something that looks like it. We want to treat a more concrete IValidation<T> as IValidation<object>. For perfectly good reasons covariance is only possible when the type parameter is only used as output, otherwise objects of the wrong type could be passed. In the given scenario however, where we know only the correct type will ever be passed, this shouldn’t be a problem.

The solution is using a proxy class which implements our less generic interface and delegates all calls to the actual instance, doing the required casts where necessary.

public class LessGenericProxy : IValidation<object>
{
    readonly IValidation<string> _toWrap;

    public LessGenericProxy( IValidation<string> toWrap )
    {
        _toWrap = toWrap;
    }

    public bool IsValid( object value )
    {
        return _toWrap.IsValid( (string)value );
    }
}

With the power of Reflection.Emit, such classes can be generated at runtime! RunSharp is a great library which makes writing Emit code feel like writing ordinary C#. It’s relatively easy (compared to using Emit) to generate the proxy class. The end result looks as follows:

object validator;  // An object known to implement IValidation
object toValidate; // The object which can be validated by using the validator.

IValidation<object> validation
    = Proxy.CreateGenericInterfaceWrapper<IValidation<object>>( validator );

validation.IsValid( toValidate ); // This works! No need to know about the type.

// Assuming the validator validates strings, this will throw an InvalidCastException.
//validation.IsValid( 10 );

Of course the proxy should be cached when used multiple times. Originally I also attempted to proxy classes instead of just interfaces by extending from them. This only works properly for virtual methods. Since non-virtual methods can’t be overridden there is no way to redirect the calls to the required inner instance.

Source code can be found in my library: Whathecode.System.Reflection.Emit.Proxy.

Beyond private accessibility

Surely you know about public, protected, internal and private. They are access modifiers often used in Object-Oriented Programming which indicate from where members can be accessed. If you’ve been following this blog you might have noticed I am a big fan of encapsulation. Previously I discussed how anonymous functions can be used to encapsulate reuseable logic within the scope of one function. This inspired me to look for more ways in which lambdas can be used for improved encapsulation.

Private access is the least permissive access level. Private members are accessible only within the body of the class or the struct in which they are declared. – msdn (.NET)

Some languages go a step further and support static locals, which allow a value to be retained from one call of a function to another with a static lifetime. The variable can only be accessed from within the function scope where it is declared. I have yet to find a language which supports the same, but with instance lifetime. Such a variable would only be visible to a single method within a single instance.

Function private?

Let’s call such a hypothetical accessibility ‘function private’. When would you use it? Consider those annoying light bulbs that burn out just as they’re turned on.


The _lifeTime variable is only relevant to the SwitchOn method. Notice how C# already offers a way to write the IsOn property in a concise way by using an auto-implemented property. This actually already limits the scope of its backing field beyond private! The auto-generated backing field can only be accessed from the getter and setter.
class LightBulb
{
    public bool IsOn
    {
      get;
      private set;
    }

    private int _lifeTime = 100;
    public void SwitchOn()
    {
        if ( _lifeTime - 1 >= 0 )
        {
            --_lifeTime;
            IsOn = true;
        }
    }

    public void SwitchOff()
    {
        IsOn = false;
    }
}

What if the following would be possible? It would encapsulate lifeTime just to the scope where it’s needed.

public void SwitchOn()
{
    private int lifeTime = 100;  // function private variable
    // private static int staticVar;  // add 'static' to create a static local

    if ( lifeTime - 1 >= 0 )
    {
        --lifeTime;
        IsOn = true;
    }
}

Lambda scope

A lambda is something peculiar. It can be used to create delegates, but how do they work? What happens behind the scenes? I asked the question on Stack Exchange, and got some really insightful answers. Relevant to this discussion are the following two points:

  • The method that backs the delegate for a given lambda is always the same.
  • The method that backs the delegate for “the same” lambda that appears lexically twice is permitted to be the same, but in practice is not the same in .NET 4.0.

The fact that every lambda has a unique backing method gives the possibility to use it as an identifier of the scope where it is defined.

Unsafe implementation

By ab(using) this behavior I was able to create the following working implementation. The Private class creates and keeps track of its instances by linking them to the delegates passed to the static constructor methods. The first time the method is called the instance is created. Subsequent calls the instance is retrieved by doing a lookup in static dictionaries based on the passed arguments.

public void SwitchOn()
{
    Private<int> lifeTime = Private<int>.Instance( () => 100, this );  // function private
    // Private<int> staticVar = Private<int>.Static( () => 0 );  // static local

    if ( lifeTime.Value - 1 >= 0 )
    {
        --lifeTime.Value;
        IsOn = true;
    }
}

Why is it unsafe?

  • This behavior is not guaranteed to be supported in later versions of .NET. As mentioned before: “The method that backs the delegate for “the same” lambda that appears lexically twice is permitted to be the same, but in practice is not the same in .NET 4.0.”
  • Objects which use the code as it is now are never garbage collected. Using weak references could solve this issue. It seems a dictionary with weak references already exists.
  • There is a small performance overhead doing the dictionary lookups. Ideally this feature would be supported at compile time.

Improved encapsulation using lambdas

Let’s take a sidestep from the more advanced runtime code generation topic (don’t worry, still more to follow later) and move to easier territory. On a blog about software design, what better topic than encapsulation.

Abstraction, information hiding, and encapsulation are very different, but highly-related, concepts. One could argue that abstraction is a technique that helps us identify which specific information should be visible, and which information should be hidden. Encapsulation is then the technique for packaging the information in such a way as to hide what should be hidden, and make visible what is intended to be visible. – Edward V. Berard

Modern object-oriented languages start to lend features from functional programming languages, like lambdas. They can be used to create anonymous functions using a very concise syntax. One of the often overlooked advantages of anonymous functions beside its conciseness is they can be used to encapsulate logic even further than a private method. When making a method private, it can be accessed by the entire scope of the class, while anonymous functions can only be accessed from the scope they are created in, or passed to.

Consider the following highly inefficient “Hello World!” implementation:

        public void HelloWorld()
        {
            string[] words = new[] { "hello", "pretty", "world" };

            Console.WriteLine(
                ComplexHelloWorldParsing( words[ 0 ] ) +
                words[ 1 ] +
                ComplexHelloWorldParsing( words[ 2 ] ) );
        }

        private static string ComplexHelloWorldParsing( string input )
        {
            string parsed = input;
            // ... plenty of complex specific parsing, only used in HelloWorld.
            return parsed;
        }

ComplexHelloWorldParsing is – and let’s assume should – only be used inside the HelloWorld method. Still it is visible to the entire class. In such a scenario, where encapsulating the specific behavior doesn’t make sense, I would use the following approach regardless of lines of code inside the delegate.

public void HelloWorld()
{
    string[] words = new[] { "hello", "pretty", "world" };

    Func complexParsing = s =>
    {
        string parsed = s;
        // ... plenty of complex method specific parsing.
        return parsed;
    };

    Console.WriteLine( complexParsing( words[ 0 ] ) + words[ 1 ] + complexParsing( words[ 2 ] ) );
}

It surprises me not all people agree on this advantage. For the same reason some professional programmers prefer to split functions just to make them smaller, some state lambdas should always be short. They state longer logic should (by a rule of thumb) be placed in a private method. If you know any pro ‘short delegate’ arguments, be sure to let me know. I started a question relating to this topic on Programmers Stack Exchange.

Creating delegates during reflection with unbound instances

Previously I discussed how to work around the variance limitations of the Delegate.CreateDelegate method. CreateDelegate has quite a few overloads, and acts differently based on the parameters which are passed. It also allows you to create open instance or closed static delegates. Without going into too much detail:

  • Closed delegate: an instance is bound to the delegate along with the method. In practice this means when calling the delegate, it will always operate on the same instance.
  • Open delegate: not bound to a specific target object until invocation time. An instance on which the method should operate is passed as first argument during invocation.

For now, I will only discuss creating open instance delegates. These allow you to invoke an instance method, while passing a first argument, specifying the instance on which it should be invoked.

Divide your programs into methods that perform one identifiable task. Keep all of the operations in a method at the same level of abstraction. – Kent Beck’s Smalltalk Best Practice Patterns

In an attempt to better follow this principle, better known as the Single Level of Abstraction (SLAP) principle, I’ve split up the CreateDelegate method into two helper methods. One to create ‘simple’ delegates, and one to create open instance delegates. Later, an additional third method could be added to create closed static delegates.

By ‘simple’ delegates I mean delegates of which the signature matches that of MethodInfo. More particularly, the method has the same amount of arguments than the delegate.

public static TDelegate CreateDelegate<TDelegate>(
    MethodInfo method,
    object instance = null,
    CreateOptions options = CreateOptions.None )
    where TDelegate : class
{ ... }

By default, this method results in a simple CreateDelegate call. Instead of having to pass the delegate type as an argument a generic approach is used, eliminating the need to cast the returned delegate. When the options argument is set to CreateOptions.Downcasting, this method behaves like the CreateDowncastingDelegate method from my previous post. Downcasts from the delegate argument types and return type to the required types of the method are generated where needed.

The helper method to create open instance delegates looks as follows:

/// <summary>
///   Creates a delegate of a specified type that represents a method
///   which can be executed on an instance passed as parameter.
/// </summary>
/// <typeparam name = "TDelegate">
///   The type for the delegate. This delegate needs at least one (first) type parameter
///   denoting the type of the instance which will be passed.
///   E.g. Action<ExampleObject, object>,
///        where ExampleObject denotes the instance type and object denotes
///        the desired type of the first parameter of the method.
/// </typeparam>
/// <param name = "method">The MethodInfo describing the method of the instance type.</param>
/// <param name = "options">Options which specify what type of delegate should be created.</param>
public static TDelegate CreateOpenInstanceDelegate<TDelegate>(
    MethodInfo method,
    CreateOptions options = CreateOptions.None )
    where TDelegate : class
    { ... }

The comments indicate clearly what TDelegate should look like. Of course, no instance can be passed, clearly specifying it needs to be specified in the delegate type. Furthermore, the passed MethodInfo is required to be an instance method. All these measures are meant to clarify this specific usage of CreateDelegate. Additionally, similar to passing CreateOptions.Downcasting to the other helper method, it allows you to break the variance safety of the ordinary CreateDelegate method.

Both helper functions can be found in DelegateHelper.cs. I also added MethodInfo extension functions to the reflection extensions, resulting in a more concise syntax.

Example usage:

MethodInfo toUpperMethod = typeof( string ).GetMethod( "ToUpper", new Type[] { } );
Func<string, string> toUpper = toUpperMethod.CreateOpenInstanceDelegate<Func<string, string>>();
string upper = toUpper( "test" ); // Will result in "TEST".

Creating delegates during reflection for unknown types

Using reflection, .NET allows you to create delegates based on MethodInfo objects. Calling a delegate is a lot more performant than using Invoke as already discussed by Jon Skeet. His results showed a delegate invocation is 600 times as fast. Plenty of reason to go through the extra effort of creating and caching a delegate. However, in a real world scenario when adding behavior to an existing class by using reflection, you’ll quickly encounter several limitations when using CreateDelegate. This post shows you where, and how to easily work around them.

In my previous article I discussed covariance and contravariance. One of the limitations of CreateDelegate is it only allows you to create delegates according to those rules. This makes perfect sense, but isn’t always desirable in practical use cases.

How would you go about creating a delegate for any method which is attributed with CallThis when the exact type of the argument is unknown? You do know the method’s signature matches Action, and only the correct type will ever be passed to the delegate.

class CallThisAttribute : Attribute { }
class SomeClass
{
    [CallThis]
    public void SomeMethod( AnyType type ) { ... }
}

...

SomeClass instance = new SomeClass();
MethodInfo methodToCall = instance.GetType().GetAttributedMethod( typeof( CallThisAttribute ) );

// The following would work, but during reflection we don't know about AnyType.
Action<AnyType> action
    = Delegate.CreateDelegate( typeof( Action<AnyType> ), instance, methodToCall );

// The following throws an ArgumentException, since the method can only be called with AnyType.
Action<object> unkownArgument
    = Delegate.CreateDelegate( typeof( Action<object> ), instance, methodToCall );

Creating this delegate isn’t impossible. This is how you would go about creating it ordinarily.

SomeClass instance = new SomeClass();
Action<object> unknownArgument = o => instance.SomeMethod( (AnyType)o );
unknownArgument( new AnyType() );  // This works ...
unknownArgument( 10 );  // ... but this will throw an InvalidCastException.

Can this downcast be generated at runtime? Yes, and the easiest approach seems to be by using expression trees. Instead of passing the type of the required delegate to create as an argument, I opted to use a generic approach. Its usage looks as follows:

Action<object> unknownArgument
    = DelegateHelper.CreateDowncastingDelegate<Action<object>>( instance, methodToCall );

Any possible delegate type can be passed. All parameters of the method, and return value are matched with those of the delegate, and casts are created where necessary.

public static TDelegate CreateDowncastingDelegate<TDelegate>( object instance, MethodInfo method )
{
    MethodInfo delegateInfo = MethodInfoFromDelegateType( typeof( TDelegate ) );

    // Create delegate original and converted arguments.
    var delegateTypes = delegateInfo.GetParameters().Select( d => d.ParameterType );
    var methodTypes = method.GetParameters().Select( m => m.ParameterType );
    var delegateArgumentExpressions
        = CreateDelegateArgumentExpressions( delegateTypes, methodTypes );

    // Create method call.
    Expression methodCall = Expression.Call(
        instance == null ? null : Expression.Constant( instance ),
        method,
        delegateArgumentExpressions.ConvertedArguments );

    // Convert return type when necessary.
    Expression convertedMethodCall
        = delegateInfo.ReturnType == method.ReturnType
              ? methodCall
              : Expression.Convert( methodCall, delegateInfo.ReturnType );

    return Expression.Lambda<TDelegate>(
        convertedMethodCall,
        delegateArgumentExpressions.OriginalArguments
        ).Compile();
}

I’m still not entirely satisfied with the name of the method. CreateDowncastingDelegate refers to the fact that downcasts are generated where necessary. Its usage looks more like an upcasted delegate however. If anyone can come up with a better name, be sure to let me know.

The entire source code can be found in the FCL Extension library. In there you can also find another method which will most likely be the subject of my next post.

Casting generic types

There, I did it! I dared to leave out covariance and contravariance from the title while that’s the very subject. Why? Because it isn’t important to know exactly what they mean and I still find those terms confusing. As a matter of fact, I won’t mention them again. If you want to know how they relate to the discussed subject, read Eric’s excellent description on the actual meaning of them. This is an introductory post to a few follow-up posts relating to the subject, so more is on the way!

You wouldn’t be reading this if you weren’t acquainted with inheritance, and probably you know about generics, so let’s start from there.

Step 1: Cats and dogs

Cats and dogs are animals.

class Animal { }
class Dog : Animal { }
class Cat : Animal { }

This allows you to safely do the following implicit casts, no big deal.

Cat cat = new Cat();
Dog dog = new Dog();
Animal animal = cat;
animal = dog;

Step 2: Pets

Cats and dogs make excellent pets, so let’s use the following generic approach.

class Pet<T>
    where T : Animal
{
    readonly T _animal;

    public Pet( T animal )
    {
        _animal = animal;
    }

    public T RunAway() { return _animal; }
    public PlayWith( T friend ) { }
}

Notice how pets can only play with animals of their own kind. Regardless of the cute picture above, this means cats and dogs don’t get along, and can’t play together. When a pet runs off, it is no longer a pet, but just an animal.

Pet<Dog> dog
  = new Pet<Dog>( new Dog() );
Pet<Cat> cat
  = new Pet<Cat>( new Cat() );
// Not possible!
// dog.PlayWith( cat );

Of course, pet dogs can still play with dogs, and pet cats with cats.

Step 3: Stray pets

The delegate representation of a method returning an object and taking no arguments is Func. Pet.RunAway is exactly such a method, returning generic type T.

Pet<Dog> dog = new Pet<Dog>( new Dog() );
Func<Dog> runAway = dog.RunAway;
Dog strayDog = runAway();

Programming is all about abstractions. When a pet runs off, is there always a need to know what kind of animal it is? The answer is no, any pet that runs off is certain to be an animal. You can easily do the following starting from .NET 4.0.

Func<Animal> runAwayDog = dog.RunAway;
Func<Animal> runAwayCat = cat.RunAway;
Animal strayAnimal = runAwayDog();
strayAnimal = runAwayCat();

Func<Animal> oldWay = () => dog.RunAway();  // Prior to .NET 4.0 this was possible using a lambda.

The same assumption can’t be made about the Pet.PlayWith method. I already demonstrated dogs can’t play with cats, so we can’t generalize dogs can play with any animal. So what is different about PlayWith? First, I need to introduce you to bulldogs.

Step 4: Bulldogs

Bulldog

class Bulldog : Dog { }
...
var bulldogPet
  = new Pet<Dog>( new Bulldog() );
var dogPet
  = new Pet<Dog>( new Dog() );

// All dogs like each other ...
dogPet.PlayWith( new Bulldog() );
bulldogPet.PlayWith( new Dog() );

A bulldog is a dog, but a dog isn’t always a bulldog. However, a bulldog is a dog, and any dog can play with any other dog, regardless of its kind. The following delegate casts are possible in .NET 4.0. Action represents a delegate with a single argument of type T but no return value.

 

Action<Dog> play = dog.PlayWith;
Action<Bulldog> bullDogPlay = play; // If dogs can get along, bulldogs can get along as well.

Action<Bulldog> oldAction = friend => play( friend );    // Prior to .NET 4.0 using a lambda.

Step 5: In and out

The conclusion is we can only make assumptions about types based on the interaction it is used in. Two interactions can be distinguished.

  • Input: A type is used as input into the class. (e.g. function argument) The type can be the same type, or any extending type.
  • Output: A type is returned from a class. (e.g. return value) The type should be the same type, meaning it is also any base type of that type.

The problem with generics is the type parameters can be used both as input and output, so an upcast to a ‘less’ generic type isn’t possible.

Pet<Dog> dog = new Pet<Dog>( new Dog() );
Pet<Cat> cat = new Pet<Cat>( new Cat() );
// Not possible ...
// Pet<Animal> dogAnimal = dog;
// Pet<Animal> catAnimal = cat;
// dogAnimal.PlayWith( catAnimal );  // Dogs don't play with cats!

But, what if you can guarantee a template argument will only be used as input, or will only be used as output? .NET 4.0 allows you to indicate this on generic interfaces and delegates by using the in and out keywords. It is these keywords which allow the aforementioned casts of the Action and Func delegates. A decision was made to leave this feature out for generic classes, since there are little useful scenarios for it.

As an example for our pets, the following is an interface where the generic type is only used as output.

interface IPet<out T>
{
    T RunAway();
}

class Pet<T> : IPet<T>
    where T : Animal
{
    ...
}
...
IPet<Dog> dog = new Pet<Dog>( new Dog() );
IPet<Animal> animal = dog;  // The generic interface can be upcasted!

Is this still not flexible enough for your purposes? Be sure to follow my next few posts which will discuss useful workarounds in scenarios where more assumptions can be made, more specifically during reflection.

CamelCase vs underscores: Scientific showdown

In the odd case that you are an experienced programmer who doesn’t have a preference over using camel case or underscores for identifiers, try making up your mind now. Try choosing independently of (language) convention, habit or type of the identifiers. If you are a Lisper and like dashes, just vote for your next favorite.

if ( thisLooksAppealing )
{
    youLikeCamelCase = true;
    votePoll( camelCaseFormatting );
}
else if ( this_looks_appealing )
{
    you_like_underscores = true;
    vote_poll( underscore_formatting );
}

Did you vote? Good! Now it’s my turn to do some work, as I will try to take you through a semi-scientific explanation to prove which formatting is best suited for programming.

I wouldn’t have written this post, if I hadn’t read Koen’s Tao of Coding. As an ex-colleague he converted me to the underscores camp. The trigger to write this post was when reading a reply on a formatting discussion.

“honestly the code is easier to read” Opinion or fact?

It inspired me to look for scientific resources. Surely, studies must have been done right? As it turns out, not too many, but I found one. But first, in case you never had this discussion, … the usual opinions, and rebuttals. If you are looking for the facts, skip to round 3.

Round 1: The opinions

Pro underscores

  • Underscores resemble natural writing the most, and thus are more readable. Spaces are simply replaced with underscores. Extreme example: isIllicitIgloo vs is_illicit_igloo.
  • Consistency with constants. Underscores are still needed in all-caps naming conventions. E.g.: THIS_IS_A_CONSTANT
  • Abbreviations could still be kept uppercase easily. E.g.: TCP_IP_connection vs tcpIpConnection
  • Classes can be kept camel case, giving a clearer difference between them and identifiers/functions. E.g.: CamelRider.ride_camel() vs CamelRider.rideCamel().

Thank you, Yossi Kreinin, for the last two points, as discussed in IHateCamelCase.

Pro CamelCase

  • Camel case is easier to type, and underscores are hard to type.
  • Camel case makes paragraphs easier to read. my_first_variable=my_second_variable-my_third_variable vs myFirstVariable=mySecondVariable-myThirdVariable
  • Camel case is shorter.
  • Camel case is used by convention in a lot of major languages and libraries. (You weren’t allowed to use this argument when voting!)

Round 2: Rebuttals

Anti underscores

  • Underscores are ugly, camel case is more elegant.

Anti CamelCase

  • Underscores aren’t that hard to type. Seriously, as a programmer it is your duty to learn blind typing with all ten fingers. Learn qwerty, and save yourself the trouble of having to use the exotic AltGr button.
  • Use whitespaces and an IDE with color coding to easily see the difference between operators and identifiers.

Round 3: The facts

When reading the abstract of the research paper, it seems science is on the camel case side.

Results indicate that camel casing leads to higher accuracy among all subjects regardless of training, and those trained in camel casing are able to recognize identifiers in the camel case style faster than identifiers in the underscore style.

Existing research

Natural language research in psychology found that replacing spaces with Latin letters, Greek letters or digits had a negative impact on reading. However, shaded boxes (similar to underscores) have essentially no effect on reading times or on recognition of individual words. Removing spaces altogether slows down reading 10-20%.

Experiment setup

Empirical study of 135 programmers and non-programmers. Subjects have to correctly identify a matching phrase (maximum of 3 words long) out of 4 similar phrases. The important variables researched:

  • Correctness: whether the subject identified the correct phrase.
  • Find time: time taken to identify the phrase.
  • Training: how being a programmer affects the performance.

Results

  1. Camel casing has a larger probability of correctness than underscores. (odds are 51.5% higher)
  2. On average, camel case took 0.42 seconds longer, which is 13.5% longer.
  3. Training has no statistically significant impact on how style influences correctness.
  4. Those with more training were quicker on identifiers in the camel case style.
  5. Training in one style, negatively impacts the find time for other styles.

The paper concludes:

Considering all four hypotheses together, it becomes evident that the camel case style leads to better all around performance once a subject is trained on this style. Training is required to quickly recognize such an identifier.

Discussion

Personally, I find the conclusion flawed for a couple of reasons.

Correctness isn’t of much importance when programming. Correctness refers to being able to correctly see the difference between similar identifiers. E.g. startTime vs startMime. This is not a common scenario when programming. Additionally, with modern IDE’s you have auto completion and indications when a written identifier doesn’t exist. This makes me believe results (1) and (3) are irrelevant. As a sidenote, I believe the correctness of camel casing is due to the slowness of the reading. When you need to take more time to read something, you will read it more accurately.

When discussing possible threats to validity they mention the following. “Essentially all training was with camel casing, it would be interesting to replicate the study with subjects trained using underscores.” Result (4) and (5) just seem unfair when taking this into account. Isn’t it obvious that people who are used to camel case are better at it. Additionally, it has a proven negative impact on the “find time” for underscores.

So, only the slowness of reading camel case (2) remains. It takes 13.5% longer on average to read a camel case identifier than an underscore identifier. Multiply this for entire code blocks, and you have my semi-scientific opinion on the war between camel case and underscores!

For those brave enough to stick around until the end, what is your opinion now? Again, try choosing independently of convention, habit or the type of identifiers.P.s.: If you still believe camel casing to be more appropriate for programming, it would be interesting to leave a comment with argumentation. 😉 I could update “Round 2: the rebuttals” to include your comments to make the article more balanced.

Update: I’ve discussed a follow-up study in a new post. They reproduced the study and measured it takes 20% longer on average to read a camel case identifier, and additionally using eye tracking they identified camel case identifiers require a higher average duration of fixations.