Friday, November 30, 2012

Teaching with Unit Tests

Unit tests, it seems, are good for more than just building robust software. I recently taught a Twisted class at Cisco, which included both a lecture and hands-on exercises. Some of the exercises were simple coding exercises: a specification the students needed to implement. One of the larger exercises, however, came with pre-written unit tests. The students needed to make all the tests pass, one by one, and the solution was finished when all the tests are done. Based on both my own observations and some feedback from the students, providing unit tests greatly improves the educational value of an exercise.

Creating The Exercise

Before explaining the benefits, I think it's worth explaining how I created the tests and what I provided the students. The exercise involved implementing a toy HTTP server, sufficient to answer simple browser requests. In order to create the tests I implemented a sample solution using test-driven development: first writing a test, then making it pass. I implemented the tests and corresponding code starting from basic features, and working up to more sophisticated features. Because I was using test-driven development, I was fairly confident (and the results from the students confirmed this) that the unit tests provided sufficient coverage to recreate my implementation. The students received only the tests, and just enough of the solution module to make it importable (some class definitions with no actual code).

In addition to unit tests, which the students were asked to solve in the given order, I also provided functional tests at appropriate points in the sequence of unit tests. For example, after implementing the minimal functionality necessary to parse and respond to a HTTP request, I provided a functional test where a real HTTP client talked to the server implementation. This often caught problems in the students implementation where their implementation was too fragile. I could have provided even more extensive unit tests, but functional tests did turn out to be a good way to catch coding issues I was unable to predict.

Once a functional test passed, the students could then run a demo server which they could point a browser at. After implementing half the tests, they were able to run a simple demo HTTP application, and when they finished all the tests, another more complex and interesting demonstration.

The Benefits to Students

  • Obvious Criteria for Success: Passing unit tests are a clear indication that the exercise had been solved correctly.
  • Guidance: Rather than having to design and structure an application with a new, unfamiliar API, the design is implicitly included. Plus, it's always clear what the next step is: make the next test pass.
  • Debuggability: Without unit tests, it's not clear how to approach a broken solution (assuming one can even recognize the brokenness). A failing unit test, on the other hand, exposes the interaction pattern with the code, and how it ought to work. A failing assertion points at what needs fixing.
  • A sense of achievement: Knowing you've gone from 2 passing tests to 3 passing tests gives a sense of accomplishment, and the knowledge that you are making progress. Without unit tests, it's hard to tell if you're getting anywhere at all; maybe the code you wrote is necessary, maybe it's isn't, but you won't really know until the full exercise is done. This is especially important for students who are taking longer to finish the exercise. Rather than feeling discouraged, as they would if they had no idea how much progress they had made, they know they are actually achieving something.

All this is based on observing the differences in students' learning and reactions between the exercises with and without unit tests. However, it's worth noting the unit test exercise was the fourth, so by that point the students did have more knowledge than when they started.

You Should Do It Too

The technique was successful enough that I am thinking that all coding exercises that teach a new library or API should include unit tests. When I have some time, I may write a post about some lessons for teachers that I learned watching students solve my exercise. And I would love to hear suggestions from people already using this technique. Test First has some information, but seems to mostly focus on teaching programming languages, as opposed to libraries or frameworks. I haven't yet had a chance to watch the videos, though. I know many others are using this technique as well.

Cancelling Deferreds, part 2

In my last post I talked about the motivation for cancellation, and how one cancels a Deferred. In this post I'm going to explain how to make a Deferred created by your code do something useful when cancelled.

Let's imagine you are implementing an HTTP client, which returns a Deferred firing with the response from the server. Cancellation is best achieved by closing the connection. In order to make cancellation do that, all you have to do is pass a function to the constructor of the Deferred (it will get called with the Deferred that is being cancelled):

class HTTPClient(Protocol):
    def request(self, method, path):
        self.resultDeferred = Deferred(
            lambda ignore: self.transport.abortConnection())
        request = b"%s %s HTTP/1.0\r\n\r\n" % (method, path)
        return self.resultDeferred

    def dataReceived(self, data):
        # ... parse HTTP response ...
        # ... eventually call self.resultDeferred.callback() ...
Now if someone calls cancel() on the Deferred returned from HTTPClient.request(), the HTTP request will be cancelled (assuming it's not too late to do so). Care should be taken not to callback() a Deferred that has already been cancelled.

As you can see, making your Deferreds cancellable is fairly simple, and the benefits are great: the ability to cancel an operation half-way in a standard manner, freeing up resources that no longer need to be used. In addition, as we will see in my next post, cancellation can be used as the basis for adding timeouts to Deferreds.

Tuesday, November 20, 2012

Cancelling Deferreds

Deferreds are Twisted's abstraction for result callbacks, encapsulating the result of some asynchronous task. For example, the result of retrieving a web page:
d = getPage("")
What if you wanted to cancel that page download? Perhaps the user hit the cancel button in the application GUI because things were taking too long. To do this, you can call the Deferred's cancel() method:
download = getPage("")
# Open a dialog that lets the user cancel the download:
dialog = ModalDialog(message="Downloading...", button="Cancel",
Cancellation of Deferreds is best effort, which is to say it may not actually cancel the underlying operation. This may be the case for a number of reasons:
  1. The Deferred doesn't know how to cancel the underlying operation.
  2. The underlying operation may have reached an uncancellable state, because some irreversible operation has been done.
  3. The Deferred may already have a result, and so there's nothing to cancel.
Calling cancel() will always succeed without an error regardless of whether or not cancellation was possible. In cases 1 and 2 the Deferred may well errback with a twisted.internet.defer.CancelledError while the underlying operation continues. Deferreds that support cancellation should document what they do when cancelled, if they are uncancellable in certain edge cases, etc..

That's cancellation from the point of view of the caller. In my next posting I will explain how the creator of the Deferred can make cancellation actually do something, e.g. stop an in-progress page download.


Future Foundries LLC is comprised of Jean-Paul Calderone and myself (Itamar Turner-Trauring). We've been core developers of Twisted for over decade, and started a company together to do Twisted and Python consulting.

Recently I've been working on training material for Twisted, including a discussion of one of my favorite techniques for making applications more robust: Deferred timeouts. Based on the slides I've created I'll start out this blog with some explanations of the Twisted API for Deferred cancellation, and then how and why you should build timeouts on top of it.