For example, if I want to sketch a grid populated with some existing data all I need to do is dragging the grid component out of the toolbar and fill in some CSV-formatted text. (Anyone experiencing drawing data-grids with Visio will know the process is not that straight-forward.) Or if I want to create a tag cloud, I just simply enter some words and Balsamiq will weight the words randomly and display them in a nice tag cloud picture. For more complex components like tree or menu, a wiki-like syntax is available to fill in the data and decorate the output (e.g. menu item separator, checked menu item etc.) This simple process of entering some plain text repeats for all kind of GUI components which need input data.

Although I am happy with what Balsamiq has to offer, I wish the software has features to allow users to draw arbitrary shapes and add custom images into the mockups. For now, users are limited to whatever GUI components and icon images provided out of the box. Regardless, I think these requirements not necessary for a majority of GUI mockups and for most of the rest I think I can resort to my modest photo-editing skills :-).

Below is a sample mockup created with Balsamiq. See more here.

The .NET framework 4.0 CTP has just been released and I think it’s a good time to explore the new features of C# 4.0. In this post, I will introduce about the following features: dynamic lookup, generics covariance and contravariance support, optional and named parameters.

1. Dynamic Lookup
If var in C# 3.0 brings about type inference for local variables with the purpose of saving a couple of key strokes, dynamic lookup adds much more to the dynamicity of the C# language. When you declare a variable with dynamic, all of its method invocations or member accesses will be resolved at runtime. For example, let’s look at the following code

public static void Main() {
    dynamic obj = “I’m statically System.String”;
    obj.NotExistingMethod(“param”);
}

In the code, we create a string object and instead of assigning it to a variable of type string, we assign it to a variable declared as dynamic. This basically instructs the compiler to not to attempt to resolve any method call or member access of the declared variable. Instead, all these resolutions will happen at runtime. That being done, the next line in which we invoke a non-existing method with an arbitrary parameter will compile just fine into CIL. At runtime, when the method call is resolved and the runtime finds that there’s no such method in the runtime type (which is a System.String), an exception is thrown. That being said, if we use a valid method then there would be no exception and the code will run fine till completion. For example:

    dynamic obj = “I’m statically System.String”;
    Console.WriteLine(obj.ToLower());

Not only that we can declare a variable as dynamic, we can also declare methods’ arguments to be so. For example:

    private static void PrintID(dynamic obj) {
        Console.WriteLine(obj.ID);
    }

    public static void Main() {
        var person = new {ID = 111, Name = "Buu"};
        PrintID(person);

        var account = new {ID = 101, Bank = "Some Bank"};
        PrintID(account);
    }

We basically create two anonymous types, for the sake of convenience, both having a property called ID and then instantiate objects out of these types. We then pass each of these two objects to the method PrintID which accepts a dynamic object and print out the ID property. The code will print out “111” and “101” respectively. And yes, we’ve just seen duck typing in action. In C#.

That example also shows an interesting usage of dynamic in regarding to anonymous types. Now, we can pass objects of anonymous types out of their declaration scope (e.g. the method) and still be able to invoke their methods or access to their members without resorting to verbose reflection code. (And in fact, whenever you see any piece of reflection code in your application and regardless of the code’s purpose, there might be an opportunity to replace it with dynamic lookup to make the code more readable.)

We are not just limited to make statically typed .NET objects become dynamic, we can use this dynamic lookup feature to conveniently interact with “actual” dynamic objects available through the Dynamic Language Runtime (DLR) included in .NET framework 4.0. In fact, one can implement such dynamic objects in C# by implementing the System.Scripting.Actions.IDynamicObject interface, which is also part of the DLR. Regardless of the actual receiver of dynamic dispatch, the usage of dynamic lookup is exactly the same from the caller’s perspective.

Some of you might wonder what code the compiler generates and whether there is any change to the CLR to support dynamic lookup or not. To answer that question, let’s look at the CIL generated from the very first code fragment of this post. (Click on the image to see the full size.)

The generated CIL is pretty verbose but a skim through it reveals two important things. Firstly, our “dynamic variable” turns out to be a plain old CLR System.Object instance. And secondly, there’s no new CIL directive or opcode to support dynamic lookup. Instead, dynamic resolution and invocation is completely handled by framework code. In fact, the above CIL is equivalent to the following C# code, but now without the dynamic keyword.

    object obj = "I'm statically System.String";
    var payload = new CSharpCallPayload(
        RuntimeBinder.GetInstance(),
        false, false, "NotExistingMethod",
        typeof(object), null);
    var callSite = CallSite<Action<CallSite, object, string>>.Create(payLoad);
    Action<CallSite, object, string> action = callSite.Target;
    action.Invoke(callSite, obj, "param");

(In fact, I simplified the code a bit so that it can fit in one method, the actual CIL generated does have a check at IL_000c which basically looks at the callSite static field of a nested class, which is generated automatically by the compiler, to see if it’s null or not before going ahead and initializing it. In order words, the callSite is cached for subsequent invocations of the same method.)

So there’s really no magic behind the dynamic keyword. What the compiler basically does is generating some payload containing the information about the invocation so that it can be made at runtime. If there’s any magic at all, it would be in the static method CallSite.Create() which uses reflection to invoke the NotExistingMethod on the given object if it’s not an instance of IDynamicObject.

Now, we know that a “dynamic object” is actually a plain old object; it should explain why method can accept dynamic parameters. It should also be of no surprise to know that dynamic lookup can also be applied to return value of an instance/static method or an instance/static field. After all, unlike the var keyword which requires the compiler to infer the exact type at compile time, the compiler in dynamic lookup scenarios can simply pick System.Object as the type.


2. Generics Covariance and Contravariance
In previous versions of C#, generic types are invariant. That is, for any two types GenericType<T> and GenericType<K>, in which T is a subclass of K or K is a subclass of T, GenericType<T> and GenericType<K> have no inheritance relationship whatsoever to each other. On the other hand, if T is a subclass of K and if C# supports covariance, GenericType<T> is also a subclass of GenericType<K> or if C# supports contravariance, GenericType<T> is a superclass of GenericType<K>.

To understand why pre-4.0 C# disallows covariance and contravariance, let’s look at some code

    var strList = new List<string>();
    List<object> objList = strList; // compile-error, with cast or not

If the code in line 2, it can be error-prone. Consider what we can write after line 2 assuming it is allowed:

    // BOOM: we're adding an arbitrary AnyObject to what, at runtime, is a list of strings
    objList.add(new AnyObject());

On the other hand, if C# supported contravariance, we could have written the following problematic code:

    var objList = new List<object>;
    objList.add(3);
    objList.add(new AnyObject());
    List<string> strList = objList; 

    // BOOM: we're getting an arbitrary object thinking it’s a string
    string element = strList.get(0);

Due to this invariant restriction, although for the good purpose, we can’t easily reuse variables and methods to respectively get assigned to or accepts various generic types. It’s somewhat unfortunately because the key thing to realize that covariance is fine as long as GenericType<T> does not have any method or member accepting arguments of type T (e.g. if we can’t add a bunch of objects into the objList, which is indeed an instance of List<string>, we’ll be safe). Besides, contravariance is just as safe if T does not appear in any return value from a member or method (e.g., if we can’t get out any string from strList, which is indeed an instance of List<object>, we’ll be safe).

Fortunately, C# 4.0 provides us an option: if a generics interface or generics delegate which has a reference type T as its type parameter and does not have any method or member that take in a parameter of type T, we can declare it to be covariant on T. On the other hand, if that interface or delegate does not have any method or member that returning T, we can declare it be contravariant on T. (Notice the emphases, only interfaces and delegates have covariance and contravariance support and the type parameter must be a reference type. On the other hand, C# arrays have been supporting covariance from the very beginning.)

Let’s look at an example. We have a Generator delegate which basically returns random instances of object of certain type (e.g. string). Its declaration is as follows:

    delegate T Generator<out T>();

Because this delegate does not take in any T as its argument, we can safely make it covariant on T. And indeed, that compiler will allow us to do so by adding the modifier out before the type parameter. However, if this delegate is declared as, say delegate T Generator<out T>(T seed) then the compiler will complain since it’s no longer safe for covariance. Now, let’s look at the usage:

    Generator<string> strGen = new Generator<string>(StringGenerator);

    // Below line is compiled now because Generator<string> is
    // subclass of Generator<object> under covariant rule
    Generator<object> objGen = strGen; 

    // Downcast is also allowed for the same reason
    strGen = (Generator<String>)objGen; 

    ...

    private string StringGenerator()
    {
        return "I'm a random string";
    }

With contravariance, you need to use the in keyword. Let’s look at an example making use of both contravariance and covariance:

    delegate K Converter<in T, out K>(T param);

This converter takes an object of type T and converts it to an object of type K (e.g. convert String to Object). Since it does not take in any K, it can safely be declared to be covariant on K. Similarly, since it exposes no T, it can safely declare to be contravariant on T. Its usage is as follows:

    var converter = new Converter<object, string>( ConvertImpl);
    Converter<string, object> string2ObjectConverter = converter;
    object result = string2ObjectConverter("A"); 

    ...

    private string ConvertImpl(object o) {
        return o.ToString();
    }

Note that although the above examples show how to declare covariance and contravariance for delegate types, it’s not different to do so with interfaces.

Before we finish with covariance and contravariance, let’s look at what code the compiler generates. After all, we know that the compiler must encode something to the CIL to denote covariant and/or contravariant generics types so that they can be consumed properly by client code. And this is what the definitions of our Generator and Converter delegates viewed from ildasm.

Do you notice anything? Those little minus and addition signs are used to denote contravariance and covariance respectively. And interesting enough, it turns out that the CLR has been supporting such CIL since the introduction of generics in .NET 2.0. Therefore, it’s possible to write CIL using covariance and contravariance under .NET 2.0+. Only by now that it’s possible to do so using C#.

A final thought about this feature, while this is a good enhancement to the language, I like the generics covariance/contravariance implementation, enabled via wildcards, in Java better for its flexibility. Anyway, let’s just be happy with it for now, we can’t have everything.


3. Optional and Named Parameters
The last two features of C# 4.0 that we’ll discuss about are optional parameters and named parameters. These features have been with VB.NET since forever and I’m glad they are finally implemented in C#.

With optional parameters, we can provide default values for methods’ and constructors’ parameters. That way, we don’t have to write a bunch of overloaded methods and constructors. For example, we can define a constructor like this:

    public Cart(int id, String name = “default cart”, double amount = 0d) {…}

Now, you can invoke this constructor with any of these calls:

   new Cart(1);
   new Cart(1, “my cart”);
   new Cart(1, “my cart”, 105.5);

How exactly does C# implement this feature? If we look at the CIL generated for this code, we’ll see there’s no magic at all. Basically the default values will be injected to the call site so that the method invocation happens normally. Here’s the CIL generated by the compiler:

Some might think that the compiler gets the default values in the source code to inject into the caller’s code. However, that will not work if a method with optional parameters is published as a library. In that case, there’s no way the compiler can get the default values to inject to the code of the library’s client. Therefore, what the compiler actually does is to encode the default parameters right into the method itself. This is what the CIL for Cart’s constructor:

Notice the .param directive which basically tells the compiler about the default values for the optional parameters. These parameters are also annotated by the [opt] attribute.

While being a great feature, there’s a caveat when using optional parameters: since the compiler inlines the default values at the caller site, change to default values at the library site won’t be reflected unless the caller is recompiled. In order words, you should consider default values of parameters as part of the published API of a method and you’d better make them right the first time.

Now, let’s say we want to call the Cart’s constructor with the ID and amount specified but not the name, how would we do that?

To avoid ambiguity (for example when two optional parameters are both strings), C# won’t allow us to simply skip a parameter like this:

    new Cart(1, 15.5d); // compile-error

An approach C# could take is to allow this syntax:

    new Cart(1, , 15.5d);

However, this looks terrible enough with just one missing parameter, much less more of them. (How do you like to read this code MethodWithManyFields(,,,15,,”param”,,)?)

For this particular situation, named parameters are excellent solution:

   new Cart(1, amount: 15.5d);

This is not the only use of named parameters though. A very important purpose of named parameters is to enhance the readability of code. Let’s say you have a class having a bunch of fields that are to be initialized in a constructor. Without name parameters, a constructor of such class will look very ugly and hard to understand without resorting to the documentation, source code, or Intellisense.

One approach is to create a builder to create instance of such class. For example:

   new BigClass.Builder().attr1(“value”).attr2(“value”)...attrN(“value”);

This will make the code more explicit about what values are to be assigned to what fields. However, the drawback is that we need to implement a builder class, which is quite a tedious task if we have to repeat that for many classes in our application.

With C# 3.0, it’s not that bad because we can use object initializer to do something like this:

    new BigClass {Attr1 = “value”, Attr2 = “value”, …, AttrN = “value”}

This looks good, but we have to define all the necessary properties for it to work, which is also a tedious thing to do if we don’t really need properties for the class or if the class itself is immutable.

With named parameters, we have a very nice solution without having to write any builder or property if we don’t want to. (Note the semicolon after the field name.)

    new BigClass{attr1: “value”, attr2: “value”, …, attrN: “value”);

    // The below does the same thing
    new BigClass{attrN: “value”, attr1: “value”, …, attr2: “value”);

Regarding the implementation of this at the CIL level, the compiler is smart enough to infer the correct argument order and simply perform a plain old method invocation.

That’s it. The key features of C# 4.0. I am personally glad that C# has come to support these. Some people said that C# has become so complex and started losing its original beauty. While I can understand that view, I think the situation is not that bad. While it’s obvious that C# has more and more constructs to supports functional and dynamic programming, the statically typed nature and the old constructs of the language are still there and no developer is forced to use the new features if they don’t need to. On the other hand, these features bring more options for those who need them and I’d rather have more options than to be handcuffed.

• The nothing-but-nails architects. These architects have hammer and thus they see everything as nails. That said, these people are very eagle to apply overkill technologies to solve even the simplest problems. If the software has web front-end, then a MVC web framework must be used. If it has to integrate with some back-end system, then an Enterprise Service Bus implementation must be deployed. Business logic? We need Enterprise Java Bean, baby. If there are 100 concurrent users, then the system should be hosted in a farm environment with 2 web servers, 2 application servers and 1 database server. And architecture design won’t stop until these architects have put all 23 GoF patterns and one dozen of enterprise architectural patterns which they just learn from a book into their Visio diagrams.
• The no-news-is-good-news architects. These architects are afraid of new things, those they often insist to be “unproven” and “risky”. Some of these architects are of same group as the nothing-but-nail guys, i.e. they are so proud of their bag of tricks that they feel no other technologies are relevant and all problems are the same. The others are way behind the technological trends that they feel threatened by new technologies and tools. The arguments these architects often use to turn down new ideas include “it will never work”, “this is not practical”, “we already has proven methods, why bother such risky solution?”, or even the discussion-stopping message “I have solved this problem before, I know what works and what do not.” I remember an occasion in which I observed a senior architect criticized heavily the design of another architect who “dared” to suggest the use of Amazon’s SimpleDB as the storage engine for their new web application. Arguments used by the former include “database over HTTP is simply unacceptable”, “we don’t need those thousands of servers in Amazon’s farm”, and “you are bringing extra risk for us by introducing another node of security”. The poor latter architect, he did not know he was suggesting something which was radically new to the mind of the former.
• The do-as-I-said architects. Unlike the no-news-is-good-news group which bashes against new ideas, this group turns down whatever ideas not coming from them. This type may not necessarily feel threatened by new things, they just feel like their ability is being challenged by ideas they did not think of from the first place. If they think using Open ESB should be used, then any suggestion about BizTalk, or a home-grown integration solution that does just the job well will be turned down. Even if they do think the latter two are better ideas. Similarly, they will turn down any “RESTful” ideas because it’s WS-* that they propose in the first place even if they understand that the REST alternative may save them bunch of implementation efforts. Just do as they said and all is good.
• The silver-bullet-obsession architects. This group stands on opposite end from the no-news-is-good-news group: they are crazy about new things. They are so interested in fads and buzzwords that they include these in any presentation they make and any design solution they propose. To these architects, it’s just cool to talk about and use new things while completely dismissing the relevancy of such things to the problem at hand. EJB? So old-fashioned, let’s use RoR. Ajax? Forget about it, RIA is the future. RIA by Flex? Silverlight with managed code is the key. ORM is the past, it should be LINQ-2-SQL all the way down, baby. And they keep spending their career looking for more silver bullets.
• The UML-only architects. If you have read my essay, The Code is the Design, you should be able to guess how bad I feel about architects who think coding is only for the wimps. Indeed, there are architects who think that if they have to write the code then there’s little distinction between them and the plain old poor developers. They think that the role is to direct, i.e. draw some diagrams and handle them out to the coders who will “realize” the design. They argue that coding is just a simple step if the design is done right. Some may go as far as stating that they don’t even need to know how to code in a language or understand the ins-and-outs of a technology before architecting a solution using those language and technology. For example I met an architect who admitted that he had not coded any single line of Java code but thought that he would do fine when architecting enterprise JEE applications. “As long as I have good Java coders to implement my design,” he concluded. A joke? Not at all. True story!

There, we have seen the 5 groups of architects who have somewhat been successfully in ruining the software architect role in the eyes of other people. It’s high time they threw away their arrogance to not look down on others’ opinions, got pass their boosterism to use the most appropriate tools for the problem at hand, and acknowledged their ignorance by continuously learning and keeping up with technological trends. And last but not least, it’s high time they started writing code again… And if the software industry has any luck, 5 or 10 years from now some developers may come across this post and think I was joking.

Unlike others, I don’t write unit test for my code. I hope my code will work right because I’m so good at coding. Even in the rare cases when it doesn’t, the QA team should be able to catch the defects easily and I will immediately know what happen to make necessary fixes. I hope I will never have to touch most of the already written code again and if I don’t have to revisit it, what would unit tests help anyway?

Unlike others, I don’t check in compilable code into the source control system very often. Instead, I code it straight for several days (or weeks) until finishing my module before checking everything in. I hope my code will not break any code of others and I hope others’ code won’t cause funny behavior in mine. Besides, my colleagues should follow the architecture design so carefully that there are no integration defects between their components and mine.

I love coding and I’m different from others. I am superior and productive. I am a hope-driven developer…

This write-up also gives me a chance to learn about some cool new features of db4o 7.2 (currently development version) such as LINQ integration, transparent activation and transparent persistence. These are really big changes from the previous version I tried (6.3). Hope that I can find some time writing about all these features. But don’t wait for me though, just go ahead and try them yourself…

• High emotional intelligence (EQ). EQ is the ability to understand, manage, and influence self’s and others’ feeling and therefore essential for effective leadership, which is mostly about working with people and making the best out of them. A person with low EQ does not know how to listen, how to respond, and how to behave properly to different people in different situations. And such person can’t be a good leader.
• Ability to articulate a vision and inspire people in that vision. Not only needs an organization a vision, even a project also needs one. A vision fosters a desired culture, establishes a compelling mission and envisions a future, enables turn-around (if one is needed), and creates a synergy among members so that they can enthusiastically and steadily move towards the shared goal and never lose focus. Without a vision, autonomy and things like that are just the management dream.
• Ability to see everyone’s strengths and put them in the right jobs. The focus on strengths, instead of weaknesses enables outstanding achievement and high staff satisfaction. Don’t give an engineer bad rating for his poor technical writing skills, give him a good rating for his outstanding problem-solving skills and hire another person to do the writing instead.
• Ability to grow and develop people. This includes things like giving proper feedbacks, setting examples, investing in suitable training, delegating well, setting smart goals, and buying them coffee (I’m not joking). If the leader gets hit by a bus, somebody in the team must be capable of taking his role. Or better yet, the team does not need a leader at all (except for someone to buy them coffee).
• Passion. You can’t do great things and inspire others to do great things if you don’t love what you do. And passion must be revealed not just through the speeches but through actions. Say you love new technologies? Show your people that you do keep up and play with new technologies, or better yet, innovate new technologies.
• Integrity, integrity, integrity. The leader must be a person his people trust. A high integrity leader treats his people with respect and is fair to all of them. (And being respectful does not mean leaders don’t sack the wrong people, those who lack of both skills and attitude to get things done and share no same value with the organization.) If there’s no integrity in the leader, all the other things are irrelevant at best. People don’t follow bosses who have no integrity. And there’s no leader without followers. Period.

Note that all these attributes do not stand in separation but are highly interrelated. You can’t grow people if you can’t see through their strengths and focus only on weaknesses. Likewise, you can’t articulate a vision and inspire your people if you don’t possess good EQ to understand and manage the differences of your people.

On the other hand, all of these attributes are learnable one can definitely learn to improve one’s capability to be an effective leader. There is no such thing as a born leader. Hardworking and continuous improvement are those that make people leaders.

If you work for a boss who displays all the above attributes, consider yourself very lucky. If you are the boss yourself and possess all these attributes, then congratulations!

Passion
You want people programmers who are really passionate about their craft and take pride on what they do. Passionate people are self-motivated; you don’t have to spend your time on preaching them on why they should add value to the team in one way or another. They are also autonomous; as long as you have given clear and challenging objectives, they will know how to get things done. Passionate programmers will spend their weekend surfing over technological blogs and feeds to keep them up to date and their evenings reading books or coding something, unrelated to the project, to sharpen their knowledge and skills. Passionate programmers are also curious, they just don’t accept the fact that there is something about this software business that they can’t understand and will go any necessary length to understand stuffs under the hood. They will “javap” Java class files to see what the bytecode instructions look like; they will decompile DLLs to see how the MS developers actually write code; they will use Fiddler to see what are actually sent back and forth between their web client and the WCF web service…

Look for passion in CVs. Do the candidates code in their free time even as simple building some “toy” projects? Are they contributors of any open-source projects? Look for passion during the interview. Asking them how they learn new things - whether they are up to date about latest technologies and trends, whether they learn things on their own initiative instead of “learning on the job”. Notice if they sound really enthusiastic when talking about technologies or display a curiosity in stuffs that they don’t know. At times, you will interview very good liars, but then, passion is among the most difficult things to be faked and I can’t think of any single mistake I’ve made on this so far.

Trustworthiness
There is nothing worse than a highly competent programmer who is dishonest. Dishonest people have excuses for almost everything. They cover their mistakes with lies and blames. They will trick anyone on everything just to get to where they want to be. They will take all praises if things go well and dismiss all the responsibilities when things go wrong. They spend office time fulfilling their personal timetable and always act as though they were the most hardworking staff. And the more competent they are, which usually mean the more authority they would assume in the project, the more damaging it will be when their mistakes are uncovered. The bottom line, dishonest people, no matter how competent, ruin the people, the project, and the organization.

On the contrary, trustworthy people will tell you and everyone else in the team about their mistakes and have a plan to avoid it from happening again. They even take note of every criticism from others in order to reflect and improve themselves. They are transparent in everything they do even if that means there are more chances of them being caught of making mistakes (“So what? Everybody makes mistake, the point is to learn from it and be a better person”, these people think.) And last but not least, trustworthy people also get thing done. They are given a task and if they say they will finish it they will finish it.

It’s quite difficult to tell if people are trustworthy or not unless you have worked with them for a while. But there are a few things that can help. Check if they really know the things they list in the CV. Check for their integrity by asking questions about their past behaviors in certain circumstances - try to ask a lot of questions and identify inconsistency in things they say. And as this is no exact science, use your instinct as well. If they display any single sign of being untrustworthiness, then unless they are so competent that you really want to give them a try, you should quickly proceed with a no-hire decision.

Teamwork
Programmers have to work well in team. You don’t need a hero who does not listen or accept almost anything said and done by others. You don’t need a hero who refuses to participate in code review just because they think their code is perfect or because they don’t want to accept any comments from “those inferior programmers”. You don’t need a hero who disregards almost every team rules and processes. These people, often cynical, never see the good things in others but can spend the whole day criticizing the bad things. In most team meetings, these people either speak a lot, leaving no space for other (”I’m the only one in this whole team that can make sense. The others had better listen to what I say”) or they don’t speak a thing (”Are these people know what they’re talking about? I’m bored to dead here.”)

On the other hand, people who have high teamwork spirit will care about everyone else in the team. And even if their colleagues are really incompetent, these people, instead of saying “I hate working with these jerks”, they will say “I think I can assist my colleagues in this particular area”. They think independently and at the same time value diverse opinions; they can work effectively on their own and at the same time care about the things others do and are always willing to give a helping hand. They not only comply with team’s discipline but also do their best to bolster it as they know discipline creates great team, or simply because they can’t just accept the idea that others have to suffer because of their sloppiness or the idea that they are too lazy to adhere to something the whole team has agreed on. They share their knowledge to their colleagues whenever they have a chance. And if you have a team wiki’s, their names will be in the top of the contributor list.

Just like trustworthiness, it’s very hard to tell if people have teamwork spirit or not unless you have a sufficient amount of time working with them. Again, instinct counts. Does this person seem to talk only about his/her achievements without any regards to the people s/he worked with? Does this person mostly use “I” instead of “We” when describing all the significant technical decisions made in his past 50-people project? What does s/he think about other job roles like tester, business analysts? Does s/he think that discipline is very necessary?

Conclusion
That’s it. Passion, trustworthiness, and teamwork spirit (plus sufficient technical ability) are all that you need. You may think these things are too obvious to be discussed at length in any blog post, but then, as simple as they are many interviewers I know often forget about them and pay too much of their time questioning the technical competence alone. The more time you spend up-front to evaluate these factors during the hiring, the better off your organization/project will be. And if some time in the middle project that you discover that the person you hired is dishonest, lacks of passion, or tends to go for one-man heroic effort, the best thing you can do for your team, and probably for that person as well, is to let him/her go very quickly.

In this blog entry, I will explore some techniques of testing the data access layer. While it won’t be very comprehensive (I think an entire book can be written just to explore in details all facets of database unit testing), I hope there is enough of topics covered for you to explore more.

But first, what is DAL?
Everyone has his or her own idea of what DAL is, but for this entry, I will stick with the following definition: DAL is a software module which encapsulates data access logic and publishes interfaces for other software module to interact with the underlying data source(s) at a high-level of abstraction (programming language objects and attributes) instead of data source’s concepts like row, columns, tables etc.

There are many ways of implementing DAL, including Active Record pattern, Data Mapper (or DAO) pattern etc. (see more in [1]), and while the techniques described in this entry will apply to all DAL implementations, the examples assume an implementation of the Data Mapper pattern because this is arguably the most frequently used data access pattern in the Java and .NET spaces.

What to test
Before writing any tests for the DAL, we should consider the following things:

• Do we want to hit the database at all or should we mock out the database communication API?
• If we have to hit the database, do we want to hit the real production version of the database or do we want to hit an in-memory lightweight database?

Some people advocate against writing unit tests which hit the database [2] because:

• A test is not really unit test if it depends on the data logic of the application, the behaviors of the database system and the underlying communication API (JDBC or ADO.NET). (That’s too much for a unit test.)
• Hitting the database slows down the test running life cycle and thus is not recommended.

According to this camp, in order to test the DAL, we should mock out the interaction with the real database system and only aim to test how the data access class responds to the caller and interacts with the database communication API like ADO.NET or JDBC regardless of how the database system really behaves. Before discussing on whether this is a good idea or not, let’s look at the some code. (Note: To run this code, you need to have the EasyMock library [3].) We basically have a DAO class, JdbcPersonDaoImpl, publishing an interface to persist objects of type Person and using JDBC to communicate with the database system. Our test class, JdbcPersonDaoImplTest, will verify whether the DAO class interact properly with JDBC.

public class JdbcPersonDaoImpl implements PersonDao {

	private DataSource dataSource;

	public void setDataSource(DataSource dataSource) {
		this.dataSource = dataSource;
	}

	public int insert(Person person) {
		try {
			Connection conn = dataSource.getConnection();
			String sql = "INSERT INTO people VALUES (?, ?)";

			// Creates a statement and asks it to return generated Ids
			PreparedStatement stmt = conn.prepareStatement(sql,
				Statement.RETURN_GENERATED_KEYS);
			stmt.setString(1, person.getName());
			stmt.setInt(2, person.getAge());

			// Executes the statement and retrieve the generated Id
			stmt.executeUpdate();
			ResultSet rs = stmt.getGeneratedKeys();
			int personId = rs.getBigDecimal(1).intValue();

			rs.close();
			stmt.close();
			conn.close();
			return personId;
		} catch (SQLException e) {
			throw new RuntimeException(e);
		}
	}
}

public class JdbcPersonDaoImplTest {

	@Test public void testInsertOnePerson() throws Exception {
		final int expectedId = 100;
		final Person person = new Person("John Doe", 30);
		final String sql = "INSERT INTO people (name, age) VALUES (?, ?)";

		// Creates a bunch of mock objects
		DataSource dataSource = createMock(DataSource.class);
		Connection connection = createMock(Connection.class);
		PreparedStatement stmt = createMock(PreparedStatement.class);
		ResultSet rs = createMock(ResultSet.class);

		// Sets expectations
		expect(dataSource.getConnection()).andReturn(connection);
		expect(connection.prepareStatement(sql,
			Statement.RETURN_GENERATED_KEYS)).andReturn(stmt);
		stmt.setString(1, person.getName());
		stmt.setInt(2, person.getAge());

		// Sets expectations
		stmt.executeUpdate());
		expect(stmt.getGeneratedKeys()).andReturn(rs);
		expect(rs.getBigDecimal(1)).andReturn(expectedId);
		rs.close();
		stmt.close();
		connection.close();

		// Turns to "action" mode
		replay(dataSource, connection, stmt, rs);

		// Creates the test fixture and feeds it the mock data source
		PersonDao dao = new JdbcPersonDaoImpl();
		dao.setDataSource(dataSource);

		// Executes and verifies results
		int personId = dao.insert(person);
		assertEquals(expectedId, personId);
		verify(dataSource, connection, stmt, rs);
	}
}

How’s that? The test code is way too complex compared with the production code and yet, the only thing it does is assuring that the production code interacts with the JDBC API properly. (And if you try to write test for a search operation which returns large result set, it is much worse.) I don’t think it is fun nor worth the effort. The complexity of the data layer has little to do with the interaction with the database communication API, but rather with the database schema and the SQL statements or stored procedures executed as part of an interaction (and if you use an ORM framework like Hibernate, the complexity of the mapping rules adds up to that matter).

In fact, I’d rather let my tests hit the real database for the SQL or stored procedures to be executed, or I would not write any test at all. (Of course, you can combine both testing the interactions and testing the real database modification, but as I said, the interaction part is too simple to be wrong while costing so much code.) Those in the real-unit-test-does-not-touch-database camp may refer to this as ‘integration’ test, not unit test, but as long as my test cases are reliable, independent, repeatable, automated, and fast (enough), I don’t care much about the terminology.

The in-memory database
To have your tests run fast, you might not want to test against your production database systems (like Oracle or MS SQL Server). Instead, you may consider an in-memory lightweight database like HSQLDB [4] or Derby [5] (see [6] for a list of in-memory databases). The nice thing about these two databases is that they have both JDBC and Hibernate support, thus it may not take a lot of effort to run your production code against these databases. (That’s not 100% certain though and it depends on what your production database is. You’d better check out the documentation of these databases.) There are a few other things you should consider when choosing this approach:

• You must make sure your schema is compatible between the test and production database. You will not want to maintain two separate sets of SQL script files. (DRY, everyone.)
• You should have your tests populate the test database’ schema on the fly to assure the automation and repeatability characteristics of good tests.

The good news is that a tool like Hibernate can populate the database schema automatically and on the fly based on the mapping files. The bad news is that sometimes you have to modify the default Hibernate configuration settings to make sure that the generated schema is compatible with your production database’ schema, otherwise, you may run into problems upon deployment.

If you use Hibernate, then testing against the in-memory database can be as simple as creating a Hibernate configuration file and a simple DAO test class which basically invokes the DAO’s interface and verifies the results (much less simpler compared to our mock-based version) like below:

hibernate.dialect=org.hibernate.dialect.HSQLDialect
hibernate.connection.driver_class=org.hsqldb.jdbcDriver
hibernate.connection.url jdbc:hsqldb:testdb/hsqldb
hibernate.connection.username=sa
hibernate.connection.password=forgetit
hibernate.hbm2ddl.auto=create-drop

@Test public void testFindAllPeople()
		throws Exception {
	// Prepares the test data
	List<Person> peopleList = new ArrayList<Person>();
	peopleList.add(new Person("John Doe", 30));
	peopleList.add(new Person("Jane Doe", 28));

	// Populates the database and flushes
	Session session = HibernateUtil.getSessionFactory().getCurrentSession();
	for (Person p : peopleList) {
		session.save(p);
	}
	session.flush();

	// Verifies the result
	PersonDao dao = new JdbcPersonDaoImpl();
	dao.setDatasource(dataSource);
	assertEquals(peopleList, dao.findAll());
}

The real database
If speed is not a big deal to you and you do not want to mess with another database and get into the compatibility headache, you can always go directly to your production database system. In that case, keep in mind a few things:

• Each developer should have her own database server on her development machine to avoid messing up with other developers’ tests.
• If software license is a problem, create a separate database instance for each developer.
• Separate the DAL test into its own suite so that you can test it as you like and don’t touch it if you are not changing any data-related stuff (DAL code, SQL, stored procedures, database schema etc.) You don’t want the possibly time-consuming unit testing process of the DAL to occur while the only thing you change is a piece of application logic.

Another thing you should consider whenever your tests hit the database, be it in-memory or production database, is that your tests must start the running with a known database state. In other words, either clean up everything before or after a test run and make sure the same data is populated in advance if necessary. And for data population purpose, consider a tool like DbUnit [7].

So, you ask “which techniques should I use?”
I think that it is not worth testing the interaction with the database communication API using mock objects if the interaction is simple (most of the time, it is simple). On the other hand, I would rather pick the test for the real production database system rather than an in-memory database to maximize the reliability of the tests while staying away from the compatibility issues. Besides, there are a few practices, as described above, to alleviate the performance and repeatability problems of hitting the real database. But of course, that’s my choice given the kinds of applications I’m doing. You need to pick the techniques that work best in your environment. And I hope I have given enough introduction for you to explore more and make you call.

References
[1]: Patterns of Enterprise Application Architecture, Martin Fowler, Addison-Wesley, 2002
[2]: http://www.artima.com/weblogs/viewpost.jsp?thread=126923
[3]: http://www.easymock.org
[4]: http://hsqldb.org
[5]: http://db.apache.org/derby
[6]: http://en.wikipedia.org/wiki/In-memory_database
[7]: http://dbunit.sourceforge.net

BTW, to those who are expecting to see part 2 of my db4o article, I am a bit overwhelmed with other stuffs lately and could not have time to start working on it; but I’ll surely do that as soon as I can.