Needless to say there have been problems along the way. Michel van Eeten, professor of Governance of Infrastructures at the Delft University of Technology, recently gave an interview to the Dutch engineering magazine De Ingenieur in which he explained how the project is suffering from an implicit choice of offloading all of the project risks (security, privacy, and so on) onto the customer. A few days ago I had a similar experience with problem offloading onto the public by this project…

Some background

Before I get into the problem itself, let me give you some background on the setup that Trans Link Systems (the executive organization behind the new system) has come up with. The idea behind the new OV-chipkaart is to replace the old Nationale Strippenkaart as a single method of payment for all public transportation services. This means that you must be able to wave your card in front of a machine at any train station, bus stop, tram stop, metro station and so on and be able to check in for travel or check out for payment (you then get charged based on distance traveled). Privacy laws in The Netherlands being what they are, each company must keep track of its own interactions with the passenger/customer, so a shared administrative database is out (meaning also that if you want to transfer from one company to another you must check out and check in again at a different sign-in point at the same station).

At the same time the government doesn’t want a monopoly on any part of the equipment, so the check-in/check-out hardware cannot be standardized to one offering by one company. Instead there is a general communications standard between the RFID chip (which is a MiFare chip supplied by NXP Semiconductors — the government agreed to a monopoly on that part) and the devices and between the devices and the public transportation operators. Trans Link Systems ties the machines together through a shared network infrastructure and also maintains a database of all transportation data (separate from the data kept by the individual companies) in order to deal with system difficulties, fraud and so on. In other words, Trans Link Systems is running a huge ESB with a monitoring system.

To reiterate, the idea behind all of this is to replace the existing Nationale Strippenkaart with the same level of service or better.

Where things started going wrong

As is often the case with huge projects that are forced on a large entity from above (a company by management or a country by the government), the problem with the OV-chipkaart project is not in the technology itself but in the interaction with the potential user base. There has been a lot to do in The Netherlands over the past few months about privacy and security involving this system. I won’t get into all of that, save to say (as mentioned before) that professor van Eeten has given a nice interview to the Ingenieur in which he points out that most of the problems are the result of some stupid choices by Trans Link Systems which boil down to dropping problems in the lap of the passengers rather than dealing with them themselves. These choices serve to lower trust in the system and the people behind it and increase the height of the hurdle to be taken in convincing people to switch.

Case in point is what happened to my mother a few days ago. My mother has had an OV-chipkaart for several months, which she ordered through the bus company (my mother travels by bus mostly, so that’s where she chose to order her card; all the cards use the same MiFare chip, but the cards themselves are supplied by many different companies and can either be anonymous or personalized). She doesn’t use it often because she is entitled to ride for free on the inner city buses, but she needs one for regional buses and the once-in-a-blue-moon train ride.

Last Monday was one of those blue-moon occasions. We took the bus to the station together and then she tried to check in for the train — and was promptly refused. Several attempts, no success. Finally I bought her an old-fashioned return ticket (say what you want about that paper stuff, it definitely works). On the way back we stopped in at the bus company to complain, who told us they couldn’t see what was up with railway payments on their machines and told us to go to the railway company (who, luckily, had an office in the same station). So we trudged over there and the railway bitch not-very-nice-person at the desk snippily informed us that my mother’s card was not equipped for the train. In order to use the train she first had to go online to order a special train subscription for not-railway-issued OV-chip cards (which was needed for the sole purpose of determining which class she wanted to travel in), which would then have to be loaded onto her card using one of the machines back at the station. And processing an order for such a subscription takes an hour, so even if you have a smartphone with Internet access and can order the thing right there it does you no good unless you want to hang around the station for an hour, thank you very much.

QoS or JPS?

Now, I’m not quite sure what I’m complaining about in this blog but it’s either a lack of QoS (Quality of Service) or an excess of JPS (Just Plain Stupid). Either way, it’s a case of what Van Eeten was talking about in his interview: offloading responsibility from the service provider onto the customer making the system difficult to use and lowering acceptance.

Sure, it doesn’t sound so bad to have to get a special subscription for the train (it’s free, you just pay for using the train) and having to wait an hour. And the reasoning even sounds reasonable (after all, the railways must know what class you will be traveling in in order to know what to charge you). But here’s the point: the public transportation providers and the government decided to change the payment system without consulting the traveling public and must therefore convince the public to switch and they’re offering a lower level of service than the old system while doing this.

Really, they are. I said it earlier and I’ll repeat it here: say what you want about that paper stuff, but it definitely works. And that’s what people are used to. You buy a single ticket and then you’re good to go. And that’s the level of service that the new system must meet at the very least. Without that the hurdle is too high for the public and acceptance will be low and grudging. After all, you must realize (which Trans Link doesn’t seem to) that the public to convince is the general public. And that general public is being asked to make changes in what it is used to. For starters, the public is being asked to move from the following process:

1. Go to stop/station.
2. Get on board.
3. Present ticket for stamping.
4. Travel.

to this one:

1. Go to stop/station.
2. Get on board.
3. Present ticket for check in.
4. Travel.
5. Present ticket for check out.

That’s only a single extra step, but for most of the general public that’s already pushing the limit of what you can ask of a person. If the card itself is difficult to handle on top of that, that’s a risk to the success of the system.

So what to do?

Compare the situation sketched above to that of the mobile phone (which I’m intimately aware of just now, having just switched ). For that you need some sort of subscription from a telecom provider and a phone (call it a card with a MiFare chip). Many providers provide some network services in addition to voice communication (like voicemail, SMS, MMS). For those you usually get pushed configurations the first time you sign in on the network. In addition most phones offer additional services (like email) which you must configure. But whether they do or not: your basic network services work the moment you put your SIM-card in the phone. Easy-peasy, Japaneasy.

The phone situation is so normal that you might have missed the point: the basic network services work immediately. No extra work or configuration required. Extra stuff like email needs configuration, but the basics are plug-and-play.

So why doesn’t the OV-chipkaart work like that? Why didn’t they make it so that you can use all public transportation in the lowest class the moment you have the card? Sure, if you want to travel in a higher class you would have to do something extra, but at least travel would be possible immediately. Why was it necessary to push the problem of getting basic service working off onto the passengers? That’s Just Plain Stupid, all the way. Not to mention WTF…

Are you kidding me?

Just as an aside, here’s an interesting question for you: how do you think tourists are supposed to get around using the new system? Tourists have to rely on the anonymous, prepaid, disposable cards that you buy at station bookstores and magazine stands. Available all over the place and no privacy concerns, for the ease of use of the hapless visitor to our country. And just to add insult to injury, those cards allow you to check in to any mode of transportation in the lowest class available automatically.

Update (05-09-2010): Since writing this blog, I have purchased an anonymous pass from the Railway Service and it has turned out that I have completely maligned them in the area of not being consistent in the treatment of personal and anonymous passes. As it turns out, you must also get a special train subscription as explained above for an anonymous pass, even if you purchase it directly from the Railway Service. So it’s equally inconvenient for everybody and tourists are just as screwed as everybody else. My apologies to the Railway Service for misrepresenting the situation.

One of the things that has gone wrong in the area of REST-the-popular-interpretation is that people think that "doing REST" is the same thing as "using HTTP" (another thing is that they think that REST is something you can do). The reason of course is simple: most people use the term REST to mean building a web site. And you use HTTP for that. And in fact I thought it was not a wholly unreasonable position because one of the constraints of the REST architectural style is such that you really would not want to use anything but HTTP as a rule.

However, today, all of a sudden, I stumbled on ultimate proof that REST really is as independent from HTTP as Fielding claims: a piece of software that uses the REST architectural style but not HTTP.

REST: the short, short version…

Before we get to this marvelous, mythical piece of software let’s have a short recap of what REST entails. Real REST that is, not the bullshit about URI mappings, JSON and all the other nonsense that REST-the-marketing-buzzword has come to mean.

REST is not a codeword for building a web site. REST is also not something that you do. REST is an architectural style, which means that it is a general idea about what your application architecture should look like with some architectural choices already made for you. In particular the REST style is a general description of application architectures that make the following choices:

• client-server: The entire application uses a client-server architecture. This principally covers the interaction between the end user and the core application, but if the application consists of multiple components they use client-server communication as well.
• stateless: All inter-component communication (including the end-user client to the application service layer) uses stateless communication. This does not mean that a component cannot have state at all (like a database), but it does mean there is no concept like a running session or a session context. It also means that all external information needed to complete a request must be in the request.
• cache-able: At the discretion of a component in the system, a response to a request may be marked as cache-able. This means that the requesting component may cache the answer for later reuse.
• uniform interface: All components in the system use the same interface, no matter what their specific functionality. This interface therefore also becomes the communication protocol of the entire system, which simplifies building communication clients. System functionality is achieved by adding a distinct, addressable resource that offers the functionality and that is controlled using the shared communication protocol.
• layered: This constraint means that you cannot see through a component that you send a request to. As long as the client doesn’t control the processing of the server, the system remains scalable.
• code-on-demand: As an optional extra a server might send code to the client for local execution.

The architectural constraints above, in the sense of a preselected set of choices, are what makes REST. Anybody see any mention of JSON? No, neither do I…

So where does HTTP come in?

If you look at the architectural constraints again, you’ll see something else that isn’t mentioned (other than JSON): HTTP. REST does not imply HTTP. So why does everybody think it does? Just because Roy Fielding helped define HTTP? Because marketing guys want to sell web site work? No, it’s because of the uniform interface constraint. That constraint specifies that each component in the system is communicated with and controlled using a single protocol. Which means a single command set. In other words, no matter what the functionality of the component, it always answers to the same command set with the same possible parameters.

The uniform interface constraint is a very horrible idea. It is a complete contradiction of the principles of domain-driven design: having a uniform interface for all your components which is also your communication protocol means that you must encode domain concepts and operations in terms of the uniform interface instead of making them explicit in a domain language. It means your attention is diverted away from the application domain and domain model and even at the level of the most basic lesson we all learned in programming ("use explicit names for your identifiers") the uniform interface is a bad idea. Nobody in his right mind would ever use a uniform interface in a multi-component application they were designing.

In point of fact, there is only one situation in which you would want to employ a uniform interface in your application: in order to allow your application to become part of a larger, generally accessible system where participation brings you more users. Think plug-in architectures like the Eclipse platform… Or the World Wide Web, which runs on HTTP. So that’s why REST has become inextricably linked with HTTP: HTTP is the protocol for a network that is an ideal use case for the REST architectural style… and it’s practically the only example of a popular network that you would want to join in order to reach a large audience.

And yet…

So it looks (for good reason) like REST and HTTP are congruent. Everybody who "does REST" uses HTTP and the HTTP-based WWW is practically the only interesting vehicle for real REST-style applications. And yet, and yet, today, in the wild, out of the blue, I came across a perfect example of that mythical beast: a REST-style application that doesn’t use HTTP as a uniform interface. So what was this application? The EZMLM mailing list manager.

EZMLM is, as said, a mailing list manager. That is, it manages a list of email addresses which are subscribed to a list (i.e. get a mail forwarded that is sent to a certain email address). In order to perform this management the EZMLM offers a number of command resources, each of which is triggered by the delivery of a message to that resource. Each command resource is individually addressable (using an email address as a universal resource identifier). My claim is that this program uses the REST architectural style. In order to prove this I must show that the architecture of EZMLM meets the constraints as explained above:

• client-server: EZMLM uses two components: a mail client (officially an SMTP user agent) and the EZMLM mail server. Like any SMTP-based email setup, this is a client-server application by definition.
• stateless: EZMLM has no concept of a session at all. Each request is somehow a mailing list management command which is atomic and no session is needed at all. The situation is somewhat deceptive because you might think that the mailing list itself is session state. However, this is not the case: EZMLM manages this list, that is true, but the list does not tie individual requests together into a session. Also, the list is not needed to make any request function correctly; all the information needed is in the mail message (usually that information consists of the contents of the FROM: header). This is true even for a command to the unsubscribe resource: the command still functions if the FROM: address is not on the list, it just cannot mutate the list (an error case is still correct processing).
  Of course the mailing list does serve as server state, especially in case of a forward command (i.e. sending a mail to the list which is distributed to all list subscribers). However, the statelessness requirement doesn’t forbid server state. The important thing to realize is that the purpose of EZMLM is to manage the mailing list and that the mailing list does not function as session state information.
• cache-able: This requirement entails that the system might decide that some server responses may be cached. The client may then use its cache instead of talking to the server. This could be done using EZMLM, especially with the EZMLM response to the "help" command (the response is a list of command email addresses recognized by the EZMLM server). In practice EZMLM won’t do this, but the requirement is not that the system MUST have responses which it considers cacheable. Also, most user agents wouldn’t be able to handle caching of mails. But again, this is not a requirement.
• uniform interface: All one server components in the system use the SMTP protocol as uniform interface.
• layered: There is only one component in the system, so this requirement is trivially met. Still, if there were more components it is hard to see how their details would not be hidden behind the SMTP interface.
• code-on-demand: EZMLM does not use this optional constraint.

Again, the essential point here is to understand that there is a difference between managing a mailing list as function of the application and using that list as session state. Another point is the definition of a component. EZMLM uses other software (particularly qmail and its shell hooks) to deliver its functionality. However, EZMLM is buildable and deployable as a single package with all functionality and must be administered and maintained as a single package. Perhaps somewhat tentatively I will consider this enough to consider EZMLM a single component.

Conclusion

Simply put: miracle of miracles, there really is software that uses a REST style architecture but doesn’t use HTTP. So yes, against all expectation it is true and REST really is independent of HTTP!

Now, Rob Pike is not nobody (in addition to being a Google principle engineer he has C and Unix credentials), plus the Go language has the Google brand name on it, so I thought it would be a good idea to check it out….

Evaluating Go

The Go programming language is described on the website as a language geared towards system programming. That is to say, it is a new(ish), general purpose programming language developed internally at Google to scratch an itch some of their developers had and they then decided to open source it.

Evidently, judging from Rob Pike’s talk at O’Reilly, it is also intended to be a simpler alternative to C++ and Java and a more robust alternative to dynamically typed languages such as Ruby, Python and such. It is this claim that I felt was particularly intriguing, so I took a look at the language this evening by reading through the language spec and briefly glancing at the tutorials.

Creak, creak…

Well, let’s address the great big elephant in the room immediately: it would seem that “simpler alternative” is Google-speak (or Pike-speak) for “forty five years old”. It’s had a spit-polish plus language features and libraries added, but there’s no way around it — Go is a rehash of the C programming language with some Pascal thrown in for good measure. Not surprising of course, given Rob’s background, but perhaps not the great linguistic revolution you would have expected.

Just to reassure you all, I’m not exaggerating: Go is essentially C, including structs, pointers and reams of different (un)signed int types. It even includes everyone’s favorite cross-platform language feature: numeric types whose size is architecture-dependent.

Spit-shine

That’s not to say that some improvements have not been made to cover up some of the glaring problems that made C and C++ hard to handle: extern is gone as a concept, as is pointer arithmetic. Plus Go has a type definition feature, so you don’t need macro’s anymore. Also, Go has some more built-in types (such as complex numbers), support for Unicode and it can insert semicolons where needed and has both escaped and unescaped strings (both borrowed from Bourne Shell). And it places the type behind the identifier instead of in front and uses := as an assignment to distinguish from equality (borrowing from Pascal).

Back from the dead

Nevertheless, not all is well with the world. Like it’s ancestors Go still relies on an ability to export named entities (including variables), meaning you can get into the global variable mess. And guess what’s back from the dead: the goto statement, in its full and glorious horror.

So what else is new?

At the core of it, Go is a component-based language. It’s main architectural feature for this is the package, which isn’t much like Java but rather borrows from the Unit found in Turbo Pascal: a collection of exported types and identifiers, with hidden implementations. Packages serve as libraries in the language and are the main strength and source of language functionality.

In the spirit of “this is your grandfather’s technology”, Go is not an object-oriented language. Instead it is fully procedural and uses the abstract data type as its main means of abstraction. Admittedly, this is a good combination with the Unit-like library. The ADT concept has been bolstered with real language support to keep ADT functions and data structures together though. Even though the main data structures are C-style structs, the language includes the concept of a method (which is a function that is sort of legoed into a data type, creating the effect of putting a function pointer into a struct in C). Methods directly associate a callable code block with a data structure, so it’s sort of like adding real behavior to your basic data type. That’s a pretty good concession to OO thinking sure enough, but encapsulation beyond the package level and inheritance are still out.

The other major feature of the language is built-in concurrency modeled on Tony Hoare’s Communicating Sequential Processes. This is a brand of concurrency that is not based around the classic “multi-process, shared single memory” model but around a “multi-process, multi-memory” model whereby processes are linked together by channels. Each process can “toss” values to another process over a channel (or “catch” values coming in across a channel). These operations block until succeeded, which allows for synchronization. The mechanism has been proven equally powerful as Dijkstra’s standard semaphore mechanism. This is once again a typical choice for someone with Rob Pike’s background as this mechanism is also included in the Bourne Shell in the form of named pipes. Of course in such a language it must also be easy to start threads or fork subprocesses and indeed this can be accomplished with the use of a single statement.

So what are we to make of this…

The Go programming language is an interesting combination of proven language features and program architectures. Rooted firmly in the Pascal/C/Bourne Shell era, it seems a bit of a museum of 1960′s language design ideas geared towards the hardware systems environment of that era’s dreams (and possibly today’s reality): heavily multiprocessed with the distinction between local and distributed processing disappeared into the background, using ADT and component-based modularity to achieve separation of concerns.

… except perhaps a bonfire?

And yet, I can’t say that I’m overly impressed with Go. Mostly I don’t see the point. Sure, its well-suited to the multiprocessing environment that we have nowadays. But its support for domain modeling and domain driven design is poor, based as it is on the ADT concept — a serious failing in a modern language if you ask me. And a language that isn’t architecture independent in a Cloud-based platform era? Come on… But most of all I don’t see the new language idea that is supposed to make this language better than all others. I don’t see a refreshing combination of functional and OO paradigms like Scala, or an “easily-typed shell over a large Java class library” like Groovy. I don’t really see what Google thinks it has accomplished with Go.

In addition to that, I don’t see proof for the claim of simplicity. Sure, it covers up the worst pointer-related problems in C/C++. Sure, multithreading is less verbose than it is in Java. But on the whole the language is no simpler than Java, as far as I’m concerned. I don’t like the lacking encapsulation, the poor man’s OSGi modularity, the way methods are tacked on to external data types rather than syntactically part of data structures. And I don’t think first-class lambda functions (popularly called closures) are all that special and I don’t think that having channels as a built-in language element is a great advantage over having them in a separate library (like C++ or Java do).

So what do I think of Go’s future? Well, it certainly has one major weapon in its arsenal: the Google logo, which still makes everything cool that it touches. So Go might get somewhere based on that. But on its own merits I can’t think of any reason to prefer it over another language.

Even though parameterized polymorphism allows you to write code that is generic with respect to the type system, sometimes you want to write code that knows which actual type(s) it has been parameterized with. This article will examine how to discover that information, as well as the discoverability limits of the Java generic type system.

Java 5.0: Generics

When Java 1.0 was introduced in 1995, it didn’t include any concept of parametric polymorphism; it was a difficult concept and considered too complicated for inclusion in the original language. So for years we as Java developers were forced to write code that relied on the supertype of the entire language (java.lang.Object) whenever we didn’t want to be specific about which type(s) our code would work with at runtime. Of course there was a major downside to this: when you formally declare that your code needs a java.lang.Object, you immediately lose the ability of the compiler to check for correct usage if you are actually going to use a more specific type. I.e. if you want to use a java.util.List to include Strings, the compiler cannot check that everybody only puts Strings into that list and not objects of a different type.

So with the advent of Java 5.0, finally parametric polymorphism was added to the language: Java was outfitted with so-called type parameters, the type equivalent of method parameters. Like a method declares formal parameters which a program can reason over in a generic way and which are only given an actual value at runtime, a class, method or interface can now take a type parameter that makes the types the code works with just as indefinite as the values of method parameters. That is, the exact type a piece of generic code will act over may not be known at programming time, but it becomes known and verifiable at compile time and therefore a safe situation at runtime. For example, consider this code:

public class Example<T>
{
     public void report(T parameter)
     {
         /* Really interesting, generic code here */
     }
}

...

Example<String> stringExample = new Example<String>();
stringExample.report("Hello World!");

Within the declaration of class Example, you as a programmer don’t know the exact type that Example will use (although you can set limits). But once a specific Example instance has been created, the compiler can check that that instance is always used with the correct type.

But now suppose that you want to know within the generic code what runtime type you’re dealing with? For example, if we modify the example above a bit like so:

public class Example<T>
{
     public void reportType()
     {
         System.out.println("This example is using type " + [what goes here?] + " for parameter T");
     }
}

Is it possible to implement the example above?

Java 5.0 and the new, generic type system

In order to answer that question, we have to delve a bit into Java 5.0′s new, generic type system — and into the representation of the type system in the Reflection API.

In previous versions of Java, the type system was rather simple: you had classes and interfaces and everything was represented in the Reflection API by instances of the java.lang.Class class. Which was arranged very conveniently, since you got all the reflective functionality for free with your regular programming work; whenever you declared a class, the Java Virtual Machine would magically arrange at runtime for a java.lang.Class instance to appear to represent the new class you had just written. And everything (even primitive types) was represented by a java.lang.Class instance, whether it was really a class, or an interface or anything else. Simple.

With the introduction of generics, the simplicity is unfortunately gone. Parametric polymorphism forces a language to have a far more extensive type system, capable of making distinctions between all sorts of different kinds of types. Before we could get away with calling everything a class; now, we have classes (like before), but we also have parameterized classes and type variables (and annotations and varargs, but let’s not care about those for now). This is because we have to be able to distinguish between a regular old class (or interface), a class that has type parameters and the type parameters themselves (which, after all, are also a kind of type in the language).

To deal with all this, Java changed its type system and the way it is represented in the Reflection API. For starters, every type in Java is now a form of type — which is represented by the new interface java.lang.reflect.Type. Since classes and interfaces are types, java.lang.Class is now a subtype of java.lang.reflect.Type. But there are more subtypes of this new interface, including java.lang.reflect.TypeVariable (which represents a type variable in a generic declaration) and java.lang.reflect.WildcardType (which represents generic wildcards used when you cannot be definite about a type). And in addition to the new java.lang.reflect.Type type, there is also java.lang.reflect.GenericDeclaration interface, which represents the declaration of a generic type (in our example above, the declaration of the Example class is a generic declaration). Since generic declarations usually declare classes and interfaces, the main implementation of the GenericDeclaration interface is also java.lang.Class.

Confused already? Good, because now it gets worse. How many types do you think you introduce to the Java language every type you declare a generic class? The correct answer is infinitely many. Of course there’s the obvious one, the generically declared class you just typed in by hand. But also every potential use with a specific type is a new type in Java. In the case of our Example class, for instance, we’ve introduced Example<T> but also (potentially) Example<String>, Example<Integer>, Example<Number> and so on. Every time you come up with an Example in which the generic parameter T has been substituted for a real type, that’s a new type in the Java language. And the JVM represents these by automagically introducing an instance of a special subtype of java.lang.reflect.Type called java.lang.reflect.ParameterizedType (which is not the parameterized type declaration that it sounds like, but rather an instance of the generic declaration with all the type varables filled in with a real value).

Type erasures and reifiable types

If the discussion above already made your mind boggle, you’re going to love what comes next. You see, when the powers that be were working on Java generics, they decided that the most important thing would be to remain backwards compatible. That is, the Example class introduced earlier should still work with Java 1.4 code once compiled. That means that all the generics stuff in Java is purely administrative data to be used by the compiler, but at runtime it all gets thrown out (you can still discover it using the Reflection API in an administrative sense, but it doesn’t affect the interaction with other classes). So, in effect, at runtime the Example class is still a class that declares a method using java.lang.Object as a formal parameter. It’s just that the compiler has done extra verification so we do not need runtime type checking.

This choice by the language designers has some consequences. For starters, it means that at compile time all generic code goes through a process called type erasure, which really means that all generic data is removed. So we forget that it is Example<T> (this becomes just Example, like in Java 1.4) and we forget that stringExample is an Example<String> (at runtime it will just be an Example).

This same choice also means that we have a distinction between what is called reifiable types and non-reifiable types. Essentially this means that in the case of some types we can completely discover at runtime what their full generic declaration was (this is called reifiable) and in other cases this is not possible.

Discovering type parameter values at runtime

So the question of discovering the type parameter values at runtime is going to center around whether we are using a reifiable type or not. And let me get right down to it: very often this is going to mean that the answer is “no, you cannot discover at runtime”. In the case of our Example class for instance, the compiler throws away all the generic information during compilation so stringExample will never know that it is working with Strings.

So when can you discover type parameter values at runtime? Well, if you have explicitly introduced a ParameterizedType (i.e. a fully defined, fully resolved form of a generic class with all the type parameters filled in). ParameterizedType information is definite, you see, and not generic. So it doesn’t get erased by the compiler. An example to clarify:

• The generic class Example<T> gets erased into Example by the compiler
• The parameterized type Example<String> stays, because it is definite at compile time

So how do you explicitly introduce a ParameterizedType? Through subtyping. If you create a definite subclass of a generic type, that means the type hierarchy at runtime must include the ParameterizedType (otherwise there is a hole in the type hierarchy). And a ParameterizedType (being a fully resolved, reifiable type) allows you to discover every detail of its being. Let’s look at this in the case of our Example. First, we introduce a subclass of Example for Strings:

public class Example<T>
{
     public void reportType()
     {
         System.out.println("This example is using type " + [what goes here?] + " for parameter T");
     }
}

public class StringExample extends Example&lt;String&gt; {}

This code will introduce three types to the language at runtime: Example (the erased version of Example<T>), StringExample and Example<String> (the String-specific, non-generic, fully reifiable supertype of StringExample). Why is Example<String> not lost as a type? Because otherwise StringExample wouldn’t have a supertype anymore. After all, Example cannot be the supertype of StringExample — it doesn’t know about Strings, only about Objects. And StringExample cannot inherit directly from Object because it has to inherit the report method from somewhere. So there has to be an intermediate type that introduced a String-specific version of the report method (and any other inherited methods). So there is this mysterious Example<String> supertype. By the way, don’t expect this type to show up as a java.lang.Class instance: it’s not a class that you can instantiate or an interface that you can cast to. But it’s there nevertheless.

So now how do we do discovery and implement the report method? As with all uses of the Reflection API, we start out by getting the java.lang.Class instance that represents StringExample. Starting in Java 5.0, Class has a new method called getGenericSuperclass(), which returns the java.lang.reflect.Type that represents the (possibly generic) supertype of the class. In the case of StringExample that is a java.lang.reflect.ParameterizedType representing Example<String>. ParameterizedTypes have an array of actual type arguments, Types representing the actual types the ParameterizedType was instantiated with. And if any of those types is a Class, that is the definite, reifiable type we are looking for. So the report method will look like this:

public class Example<T>
{
     public void reportType()
     {
         Class thisClass = getClass();
         // Since we KNOW this must be a ParameterizedType, we can cast
         ParameterizedType pType = (ParameterizedType)thisClass.getGenericSuperclass();
         Type firstType = pType.getActualTypeArguments()[0];
         Class whatWeWant = (Class)firstType;
         System.out.println("This example is using type " + whatWeWant.getName() + " for parameter T");
     }
}

public class StringExample extends Example<String> {}

Of course, normally you would have to check whether firstType is really a java.lang.Class.

Conclusion

In this article we have discussed the changes to the Java 5.0 type system with respect to genericity and with an eye towards runtime discovery of type parameter values. This has lead us to review the new concept of Type in the type system and the many subtypes thereof. It has also lead us to examine the differences and causes of differences between reifiable and non-reifiable types. Finally, we have distinguished when type variable values are discoverable and when they are not, as well as how their values can be discovered.

As with many a long and winding trail, this one started this morning with what seemed like a short hike across a small and uncomplicated footpath. You see, of late I have found myself making more and more use of the different features of Outlook (something to do with becoming more senior in my company…). Among others, I’ve found that I’ve started using the little flags to mark mails that I have to get back to. But I receive so many mails nowadays that even in a single day the flags get snowed under and pushed off the list. Of course I could just sort by flag status, but I prefer to have today’s mails at the top (rather than the flags and ticks from two weeks ago). So I had myself a bright idea this morning: I’ll just change the grouping of my mails to group by received date and then flag status within the date groups. Walk in the park, right?

Oh mother….

First strike: the very simple idea

Okay, to recap, I wanted to end up with the mails in my inbox grouped first by date of receipt, then by flag status. Now, as it happens, if you do nothing to your Inbox (or any mail folder), Outlook 2007 sorts mails by date by default. You can even see this: just right-click the column headers in Outlook and select Group Box. You’ll see a tree display of grouping fields.

So, simple idea: just drag the Flag Status column into this Group Box to add it as a grouping to the Date. Unfortunately though, Date is an implicit rather than real property that Outlook groups by if there is nothing else and you cannot add any sub grouping to it. You can replace Date with anything, but you cannot add to it. Scratch that idea.

Second strike: a slightly less simple idea

So, the implicit Date as main grouping is out. No biggie, because the Receipt time is included in each mail as a property by Outlook. You even see it as a column in the default layout of every mailbox. Quick job to select that instead of the Date, then subgroup by Flag Status.

The result is, unfortunately, unusable. You see, the Receipt column includes the time as well as the date, even if you format the column to include only the date (the column format is really just for display). Almost no two mails arrive within the same minute. Group on Receipt time and you’ll end up with a million groups consisting of just one mail (maybe two every now and then). So that was that idea up the spout as well.

Third strike: some formula magic

Okay, so we cannot sort on Receipt time because it is too detailed. And there is no date-only column. But Outlook has a neat little party trick: user-defined fields (UDFs). You can define mail items to have custom properties and you can even associate a VB formula with those properties. So here’s our solution: create a custom property which will contain the Receipt time with the time part stripped off. Then use that as the main group selector.

Creating a UDF in Outlook 2007 is easy: right-click the column headers and select Field Chooser. From the drop-down, select “User-defined fields in Inbox”. Then hit the New button and select Formula as type. Fill in a name for the field and in the Formula box fill in

DateValue( [Received] )

Now right-click the column headers again and select “Customize current view”. Hit the “Group by” button. Uncheck the checkbox and select “User-defined fields in Inbox” from the drop-down at the bottom. Now, in the first drop-down for grouping field, select your new field and…. snag. You cannot select the field.

You see, you cannot group by formula UDFs in Outlook. Probably because VB is typeless in principle, so Outlook cannot predict the type of your field and so cannot compare the values reliably. So that idea is out as well.

The long way around

So that means there’s only one way to do this: add a custom property to each mail in the Inbox with the date part of the receipt time. And since it has to be typed field, it has to be added programmatically. So the long road means delving into the guts of VBA for Outlook 2007.

Since we’re going into programming mode here, it is necessary to think a little about what must be done. We want to add a custom date property to each mail in the inbox. But there are two kinds of mails in the inbox:

0. Mails that are already there, which require a one-time loop to add the property.
1. Mails which will come in later and must have the property added as they arrive; this will require event handling.

In both cases we will have to perform the following, general steps:

0. Get hold of the mail M to be augmented.
1. Get M’s receipt time R.
2. Figure out the date part D_R of R.
3. Create a custom property EmailDate for M, with value D_R.
4. Save M with its new property EmailDate.

Visual Basic for Applications in Outlook

Let’s start with some basics about Outlook Basic (ha, ha, I make a funny). Outlook VBA all lives in one large VBA project file called VBAProject.OTM, which contains one project called Project1. Everything has to be added there. Project1 itself consists of a sort of main module called ThisOutlookSession. This is the place to put simple macro’s and the entry points to larger programs.

VBA for Outlook is essentially the same VBA you find in any Microsoft application: a VBA editor (hit Alt+F11 in Outlook) which supports a subset of Visual Basic with some built-in functions and an application-specific object model. In the case of VBA for Outlook this model models things like Folders, MailItems, CalendarItems and so on.

Like all VBA languages, VBA for Outlook allows you to create special variables which can hold object instances that are aware of application events. For example, if you declare a variable like so:

Dim WithEvents inboxItems As Outlook.Items

and then instantiate inboxItems to be the list of items in the inbox, then inboxItems will be aware of any events that occur in the inbox. You can declare event handlers for any interesting events you want to respond to.

Dealing with existing messages

Let’s start simply with the existing messages in the Inbox. To deal with those we need a run-once loop and no event handling. In order to carry out the process outlined above, we will have to get hold of the folder object that represents the inbox in Outlook. We can then loop over the contents of this folder.

While setting the process for existing messages, we will be dealing with mail messages, which are Outlook.MailItem objects. We will be adding a custom property, which is an Outlook.UserProperty object. And we have to deal with the inbox, which is a mail folder — an Outlook.MAPIFolder object.

Getting hold of the Inbox folder is done by getting the default inbox folder from Outlook’s MAPI namespace — this is an object that we can get from the Application object. The Application object is present in every VBA language and represents the application (in this case, Outlook). The MAPI namespace has a getDefaultFolder method which takes a convenient constant to indicate we want the Inbox. After that we can loop over the objects in the Inboxes Items list to set the properties. The result looks like this:

Private Sub SetInitialProperties()
  Dim ns As Outlook.NameSpace
  Dim fld As Outlook.MAPIFolder
  Dim objUnknown As Object
  Dim mail As Outlook.MailItem
  Dim prop As Outlook.UserProperty

  Set ns = Application.GetNamespace("MAPI")
  Set fld = ns.GetDefaultFolder(olFolderInbox)

  For Each objUnknown In fld.Items
    If TypeOf objUnknown Is MailItem Then
      Set mail = objUnknown
      Set prop = mail.UserProperties.Add("EmailDate", olDateTime, True)
      prop.Value = DateValue(mail.ReceivedTime)
      mail.Save
      Set prop = Nothing
      Set mail = Nothing
    End If
  Next

  Set objUnknown = Nothing
  Set fld = Nothing
  Set ns = Nothing
End Sub

As you can see, adding a custom property is pretty straightforward; there are constants to indicate the type (which is why we are using VBA, of course). Getting the date is a matter of manipulating the existing ReceivedTime property of each mail a little. It is worth noting in the loop that there is a check to see if each item is really a MailItem — the Inbox may also contain sub folders, you see.

Next, hit the Run button to run the routine. Then check in Outlook (right-click the column headers, select the Field Chooser and select the UDF’s from the drop-down; drag the new field onto the Inbox columns). The new field contains the date of the mails, but the time midnight (so no more minutes to screw up the grouping). Try grouping by the new column — this time it works, since the new column has a type.

Dealing with incoming mail

In order to set the same property on incoming mail, we need an event handler. As explained above, we can create a special variable that will be made aware of events on any specific object (like the list of items in the Inbox). In fact, that list is particularly useful to monitor since it broadcasts an event every time an item is added. With that information, the actual event handler is pretty straightforward:

Private Sub inboxItems_ItemAdd(ByVal Item As Object)
  Dim prop As Outlook.UserProperty
  Dim mail As Outlook.MailItem

  If TypeOf Item Is MailItem Then
    Set mail = Item
    Set prop = mail.UserProperties.Add("EmailDate", olDateTime, True)
    prop.Value = DateValue(mail.ReceivedTime)
    mail.Save
    Set prop = Nothing
    Set mail = Nothing
  End If
End Sub

Again, note the check for whether an item is a MailItem and not a new sub folder.

As you can see, the VBA convention is to have the name of the event handler be <variable>_<event>(<parameters>). This is also how the VBA interpreter links events to methods (or subroutines in this case).

Now, the event handler is pretty simple. But it does depend on the variable inboxItems being initialized to refer to the list of items in the inbox. So how is this accomplished?

Initializing the inbox variable

Obviously, since Outlook does not provide a database for its VBA macros, there is no such thing as a persistent variable. We will have to redefine the inboxItems variable each time Outlook starts. You guessed it: we need another event handler, written the same way as the earlier one (a subroutine referring to a variable). But isn’t that a chicken-and-the-egg problem? How do we initialize that variable? Luckily, as remarked before, there is one variable which is always implicitly available: the Application variable. And that represents Outlook which, lucky for us, is exactly what we need (we want to listen for application starts of Outlook). So our primal event handler will look like this:

Private Sub Application_Startup()
  Dim ns As Outlook.NameSpace
  Dim fld As Outlook.MAPIFolder

  Set ns = Application.GetNamespace("MAPI")
  Set fld = ns.GetDefaultFolder(olFolderInbox)
  Set inboxItems = fld.Items

  Set ns = Nothing
  Set fld = Nothing
End Sub

Startup is, of course, the event fired when Outlook starts.

Testing the event handler

The simple way to test the event is to run the Application_Startup() subroutine using the Outlook VBA tool to set the system up. Hit Alt-F11, open the ThisOutlookSession module (which contains the code described above), put the cursor in the Application_Startup() routine and hit the Run button. Then go back to Outlook and send yourself a mail. Once it arrives, it will have the new property automatically set.

Just when you think you’re there…

If you’ve been following along, there’s another test you can do (and which I did): close Outlook and start it up again. Then send yourself another mail. If you’re running Outlook 2003 or 2007, the new mail probably will not have the new UDF set.

So what went wrong? We created the event handler, the Outlook startup event handler to set everything up and tested it to see it working? Well, some debugging will tell you almost immediately that the Outlook startup event handler is not called. And the reason, it eventually occurred to me, is macro security. You see, by default Outlook will not run unsigned macros. Even macros defined in its own macro module.

You can get around this easily by going to the security settings for macros (in Outlook 2007 it’s called the TrustCenter) and selecting to run unsigned macros with a warning. But a better idea is to create a self-signed certificate and use that to sign your macro module.

To create the certificate, you need a tool called SelfCert.exe, which is installed with Office in the Office install directory (try c:\program files\Microsoft office\office or c:\program files\Microsoft office\office12). Run the program, fill in a name for your certificate and click OK. Then go to the Outlook VBA editor and select Digital Signature from the Tools menu. Click Choose and select your certificate. Click Ok and then save the module.

Now restart Outlook. It will give you a warning about macros, but if you select “Enable macros” everything will work. You can elect to trust your own certificate in the same dialog, which will eliminate the warning.

Finally…

So, finally, after all that, my Inbox is now grouped by date and then flag status. That took some doing…..

But perhaps someone else will find the experience useful.

Sources

The following forum archive provided me some valuable insights into how to get all of this working: www.eggheadcafe.com/forumarchives/outlook/Oct2005/post24967825.asp

Bad Astronomy

Philip C. Plait. Wiley 2002, Paperback, 288 pages, $8.38

Reading through the book I came to chapter 14, which deals with the doomsday advocates who swore up and down that the world was going to come to an end on May 5, 2000 due to a planetary alignment (more or less) of Mercury, Venus, Mars, Jupiter and Saturn. It was quite funny to read Plait’s description of why this neither would nor did cause catastrophe for our planet. But more than that, the chapter got me thinking about how bad science is all around us and affects us in sometimes rather drastic ways.

In the case of Plait’s example nothing really very drastic happened to anybody (except a few people who were duped into buying bogus survival gear to help them survive the new ice age that the planetary alignment was supposed to drop the planet into). However, not everything that is misunderstood is so innocent. Take for example a court case in the Netherlands (this was quite a long time ago, to be honest):

In this particular case a bridge guard was being sued by a barge captain because the captain felt the bridge guard didn’t open the bridge often enough daily. The bridge was too low for the barge to pass under (by a meter) and the frequent delays cost the captain money. So he sued to get the bridge opened more often. The judge in this case was a long-standing lord of the bench, properly educated in languages and law (as most judges), but not so much in the sciences. His ruling was that the bridge guard could not be forced to alter his ways and that instead, the captain should pay (out of his own pocket) to have the river bed under the bridge dug out an extra meter. It took testimony from a professor of physics of the University of Delft to explain to the judge that that is not how it works.

Lawyers (and related persons) are particularly good at bad science, it seems. And not just physics, but also mathematics and logic. A former professor of mine (a computer scientist, philosopher and master of law) told us some particularly nice anecdotes about the brilliance of Dutch traffic law, for instance. Dutch traffic law specifies that there are priority vehicles (ambulances, police cars, etc.) which have priority in all traffic situations. Dutch traffic law also specifies that traffic signs (signs, lights, arrows painted on the road, etc.) have priority over traffic rules. Makes sense so far, right? Except, my professor pointed out, the law was written by lawyers. Who encoded the two rules mentioned into two separate and completely unrelated articles like so:

A. One must always give precedence to priority vehicles.

B. Traffic signs have priority over traffic rules.

Taken like that, the law says that priority vehicles must stop for red lights. We are all grateful that they disobey the law constantly.

Recently, I got embroiled in an even more far reaching example of bad logic and the law. On some forum somewhere I got into an argument with someone (whom I suspect is a first-year law student) about the Dutch constitution. Our constitution is a pretty typical constitutional document which describes the basic organization of country and government, as well as the civil rights of the citizens and other inhabitants. Perhaps a little unusual (but quite well thought-out, I feel), our constitution recognizes that not every right can or should be absolute. Several of our civil rights are therefore worded in the following structure:

For example, our constitution guarantees the right to free speech; but the constitution also leaves room for the law to set limits (such as forbidding insults to whole groups within society).

The suspected law student, in a bit of brilliant logical derivation, arrived at the conclusion that it is possible for the law in our country to contradict the constitution and therefore that our constitution as a whole is meaningless. That is (to stick with the example above), if it is forbidden to insult all Jews/Christians/Muslims in the country, then the law violates the free speech article and since the constitution is not enforced over the law it means nothing. I’ll admit that the truth is a little subtle: the constitution sets a general rule and allows the law to create exceptions. So the law creates an exception (or violation) of the general rule, but not of the constitution. But still, you’d think that even a law student (a lawyer to be) should be able to spot that he’s claiming that a law whose existence is explicitly allowed by the constitution is a violation of that constitution…

Bad science, I fear, is all around us all the time. Especially since the environment has started playing a major role in decision making. From the greenhouse effect down to whether or not you are poisoning the environment by dumping non-biodegradable plastic in it (you’re not, by the way — non-biodegradable plastic isn’t biodegradable, so it doesn’t interact with the environment if dumped in nature somewhere), bad science pervades life. Two months ago, in The Netherlands, there was a great commotion is some village about plans to store carbon dioxide gas in an old natural gas pit which has been depleted by the national gas seller. Now, this was a porous rock stratum at over a kilometer depth in which nature has stored flammable gas for millions of years, under pressure, without a single leak or explosion. But for some reason the inhabitants (who had been living, technically, on a bomb for centuries before the gas was pumped out) are now scared of living on top of inert, non-flammable gas that is to be stored in the same stratum under less pressure than the gas was under.

Bad science is often funny to read about. But equally as often it is no joke at all, as it leads to bad decisions. When people misunderstand science and especially when bad people are creating bad science in order to make people make bad decisions. So a valid question arises: where are the scientific and engineering communities amidst this storm of bad science? Why don’t they have a more active role in society and why doesn’t society ask them to have a more active part?

In this country, we have several professional societies for scientists and engineers. I’m even a member of one of them (KIvI/NIRIA). Now, it’s not that they don’t try. Really, they do. KIvI/NIRIA, the ONRI, the VSNU, all sorts of others, they all feel the need to play a more active role in society, to influence decision making and to try and be voices of reason amidst the insanity. But they do it in the way engineers have become accustomed to over the past forty years; silently, in the background, trying to be the quiet advisor in the ear of the board. And so they often miss; they get drowned out in the melee, overruled by the opinions of the crowd that are based on nothing that makes any sense. And then, instead of trying to push bask, they submit to the nonsense because the decision has been made and once the decision has been made you no longer disagree. And that may be the worst science of all….

Unfortunately, with the new laptop I was also downgraded: from Windows XP to Windows Vista and from Office 2003 to Office 2007. And no, those are not typos; I’ve been given new software with my new laptop and I hate every little bit of it. Vista is designed to kill your productivity and break your existing software. And even if you have software like Office (guaranteed to work due to the close ties between development teams), they make sure the new version is so bad that you can’t use that either.

Among my favorite bugs, failures and stupidies so far are the following:

The LAN mystery

At my current project, I used to connect to the corporate LAN. Nothing spectacular there: I just plugged in the network cable and set up the proxy script, filled in my username and password when prompted, done. With Vista, it doesn’t work anymore. Not even in Opera, which doesn’t even share network access configuration with IE7! The damn thing simply ignores my proxy server settings, doesn’t prompt me for a password and then cannot connect properly to the internal systems. Is it a new browser version? No, because I used the same browsers on XP. It’s Vista.

But don’t think I’m locked out though — my wireless can connect. Slower than LAN, but still. Mind, I don’t think it works properly; I think I’ve just lucked out in that I have to enter PEAP-authentication details for the wireless and those settings get passed on to the proxy. Because I don’t think the wireless stack is picking up the proxy settings correctly either.

Opera and Media Player

On my old laptop, I used Opera 9.64 and Windows Media Player 11. Worked together just fine. In Vista, I suddenly need a special plugin to enable the same combination. Sorry, but there’s no way you get that without doing it on purpose. Come on, European Commission, why do you keep screwing around with these guys? The maximum fine is 10% of their gross, well over $700 million a pop!

Ribbons…

So after all that joy, I needed Word today. Word 2007, my first application-related WTF moment on my new laptop. Who the hell came up with that damned ribbon? I can’t find ANYTHING anymore! My normal working day includes IDEs, text editors and browsers. Mostly, I don’t need Office. So when I DO need it, I don’t want to be tossed into a bloody maze of having to find out where they put everything. Thankfully there are tools available online that send the ribbons back to the hell they came from and give you back the menus intended for human beings.

I am NOT a moron

While trying to work out why my network connections weren’t working, I stumbled on this great new feature: not only have all the real settings been hidden beneath at least five layers of useless windows, but now you have to give yourself permission to access most of them. I AM NOT A MORON! I DON’T NEED A NANNY, NOR A KINDERGARTEN COMPUTER! I HAVE AN ADMINISTRATOR ACCOUNT SO JUST LET ME INTO THE FUCKING SETTINGS!

God, I want a Linux distro on my laptop.

And finally… shadow files

This one was the final straw that convinced me to spew my wrath in a blog post. One of the more brilliant innovations in Vista is a built-in revision system called Shadow Files. It makes regular copies of all your files so you can rollback any mistakes. Everybody who has ever used Subversion will understand the benefits. But of course, Microsoft got it wrong: After I got done installing all my software, end of day Tuesday, I had 105 Gb free (out of 147). By midday today, it was down to 91,4 Gb. Turned off the miserable Shadow Files and shot back up to 113 Gb. 22 Gb just for useless revisions. So here’s my top tip for the day:

TURN THE DAMN THING OFF

Seriously. It devours your harddrive, for a feature that you don’t need. No, I’m not kidding; you’ve lived without it for years for the stuff you never touch (most of your system) and if you have files for which you really DO have to save revisions, Subversion does a perfect and controllable job. You’ll be fine without it, so save your harddrive and the aggravation.

Is there any salvation?

Well, not for me at least. I’ve asked and I cannot get my old laptop back. Nor can I install WinXP on my own, or any other OS. So I’m basically screwed until the next system upgrade. But at least I can offer everybody else this wisdom: don’t downgrade to Vista. If they offer you a new system, resist. It’s just not worth the gray hair.

Enjoy!