Enough of the fun part, now seriously. JTeam has outgrown it’s name. The past year a lot has changed and a new name is the next logical step. I cannot explain this better than my CEO Steven Schuurman. If you are interested you can read his announcement here:

http://blog.dutchworks.nl/2011/10/10/hot-off-the-press-as-of-today-“jteam”-will-be-known-as-“dutchworks”/

Head over to our new website: http://www.dutchworks.nl. Help us spread the word.

Hope to see you back when we write our next technical blog.

Continue reading… (full article published on blog.jteam.nl)

0.7 offers a large number of new features, such as support for Sagas, Event versioning, Spring Namespace Configuration, and many others, which I will describe in more detail.

Continue reading… 
(the entire post is published on blog.jteam.nl)

Needless to say there have been problems along the way. Michel van Eeten, professor of Governance of Infrastructures at the Delft University of Technology, recently gave an interview to the Dutch engineering magazine De Ingenieur in which he explained how the project is suffering from an implicit choice of offloading all of the project risks (security, privacy, and so on) onto the customer. A few days ago I had a similar experience with problem offloading onto the public by this project…

Some background

Before I get into the problem itself, let me give you some background on the setup that Trans Link Systems (the executive organization behind the new system) has come up with. The idea behind the new OV-chipkaart is to replace the old Nationale Strippenkaart as a single method of payment for all public transportation services. This means that you must be able to wave your card in front of a machine at any train station, bus stop, tram stop, metro station and so on and be able to check in for travel or check out for payment (you then get charged based on distance traveled). Privacy laws in The Netherlands being what they are, each company must keep track of its own interactions with the passenger/customer, so a shared administrative database is out (meaning also that if you want to transfer from one company to another you must check out and check in again at a different sign-in point at the same station).

At the same time the government doesn’t want a monopoly on any part of the equipment, so the check-in/check-out hardware cannot be standardized to one offering by one company. Instead there is a general communications standard between the RFID chip (which is a MiFare chip supplied by NXP Semiconductors — the government agreed to a monopoly on that part) and the devices and between the devices and the public transportation operators. Trans Link Systems ties the machines together through a shared network infrastructure and also maintains a database of all transportation data (separate from the data kept by the individual companies) in order to deal with system difficulties, fraud and so on. In other words, Trans Link Systems is running a huge ESB with a monitoring system.

To reiterate, the idea behind all of this is to replace the existing Nationale Strippenkaart with the same level of service or better.

Where things started going wrong

As is often the case with huge projects that are forced on a large entity from above (a company by management or a country by the government), the problem with the OV-chipkaart project is not in the technology itself but in the interaction with the potential user base. There has been a lot to do in The Netherlands over the past few months about privacy and security involving this system. I won’t get into all of that, save to say (as mentioned before) that professor van Eeten has given a nice interview to the Ingenieur in which he points out that most of the problems are the result of some stupid choices by Trans Link Systems which boil down to dropping problems in the lap of the passengers rather than dealing with them themselves. These choices serve to lower trust in the system and the people behind it and increase the height of the hurdle to be taken in convincing people to switch.

Case in point is what happened to my mother a few days ago. My mother has had an OV-chipkaart for several months, which she ordered through the bus company (my mother travels by bus mostly, so that’s where she chose to order her card; all the cards use the same MiFare chip, but the cards themselves are supplied by many different companies and can either be anonymous or personalized). She doesn’t use it often because she is entitled to ride for free on the inner city buses, but she needs one for regional buses and the once-in-a-blue-moon train ride.

Last Monday was one of those blue-moon occasions. We took the bus to the station together and then she tried to check in for the train — and was promptly refused. Several attempts, no success. Finally I bought her an old-fashioned return ticket (say what you want about that paper stuff, it definitely works). On the way back we stopped in at the bus company to complain, who told us they couldn’t see what was up with railway payments on their machines and told us to go to the railway company (who, luckily, had an office in the same station). So we trudged over there and the railway bitch not-very-nice-person at the desk snippily informed us that my mother’s card was not equipped for the train. In order to use the train she first had to go online to order a special train subscription for not-railway-issued OV-chip cards (which was needed for the sole purpose of determining which class she wanted to travel in), which would then have to be loaded onto her card using one of the machines back at the station. And processing an order for such a subscription takes an hour, so even if you have a smartphone with Internet access and can order the thing right there it does you no good unless you want to hang around the station for an hour, thank you very much.

QoS or JPS?

Now, I’m not quite sure what I’m complaining about in this blog but it’s either a lack of QoS (Quality of Service) or an excess of JPS (Just Plain Stupid). Either way, it’s a case of what Van Eeten was talking about in his interview: offloading responsibility from the service provider onto the customer making the system difficult to use and lowering acceptance.

Sure, it doesn’t sound so bad to have to get a special subscription for the train (it’s free, you just pay for using the train) and having to wait an hour. And the reasoning even sounds reasonable (after all, the railways must know what class you will be traveling in in order to know what to charge you). But here’s the point: the public transportation providers and the government decided to change the payment system without consulting the traveling public and must therefore convince the public to switch and they’re offering a lower level of service than the old system while doing this.

Really, they are. I said it earlier and I’ll repeat it here: say what you want about that paper stuff, but it definitely works. And that’s what people are used to. You buy a single ticket and then you’re good to go. And that’s the level of service that the new system must meet at the very least. Without that the hurdle is too high for the public and acceptance will be low and grudging. After all, you must realize (which Trans Link doesn’t seem to) that the public to convince is the general public. And that general public is being asked to make changes in what it is used to. For starters, the public is being asked to move from the following process:

1. Go to stop/station.
2. Get on board.
3. Present ticket for stamping.
4. Travel.

to this one:

1. Go to stop/station.
2. Get on board.
3. Present ticket for check in.
4. Travel.
5. Present ticket for check out.

That’s only a single extra step, but for most of the general public that’s already pushing the limit of what you can ask of a person. If the card itself is difficult to handle on top of that, that’s a risk to the success of the system.

So what to do?

Compare the situation sketched above to that of the mobile phone (which I’m intimately aware of just now, having just switched ). For that you need some sort of subscription from a telecom provider and a phone (call it a card with a MiFare chip). Many providers provide some network services in addition to voice communication (like voicemail, SMS, MMS). For those you usually get pushed configurations the first time you sign in on the network. In addition most phones offer additional services (like email) which you must configure. But whether they do or not: your basic network services work the moment you put your SIM-card in the phone. Easy-peasy, Japaneasy.

The phone situation is so normal that you might have missed the point: the basic network services work immediately. No extra work or configuration required. Extra stuff like email needs configuration, but the basics are plug-and-play.

So why doesn’t the OV-chipkaart work like that? Why didn’t they make it so that you can use all public transportation in the lowest class the moment you have the card? Sure, if you want to travel in a higher class you would have to do something extra, but at least travel would be possible immediately. Why was it necessary to push the problem of getting basic service working off onto the passengers? That’s Just Plain Stupid, all the way. Not to mention WTF…

Are you kidding me?

Just as an aside, here’s an interesting question for you: how do you think tourists are supposed to get around using the new system? Tourists have to rely on the anonymous, prepaid, disposable cards that you buy at station bookstores and magazine stands. Available all over the place and no privacy concerns, for the ease of use of the hapless visitor to our country. And just to add insult to injury, those cards allow you to check in to any mode of transportation in the lowest class available automatically.

Update (05-09-2010): Since writing this blog, I have purchased an anonymous pass from the Railway Service and it has turned out that I have completely maligned them in the area of not being consistent in the treatment of personal and anonymous passes. As it turns out, you must also get a special train subscription as explained above for an anonymous pass, even if you purchase it directly from the Railway Service. So it’s equally inconvenient for everybody and tourists are just as screwed as everybody else. My apologies to the Railway Service for misrepresenting the situation.

As with many a long and winding trail, this one started this morning with what seemed like a short hike across a small and uncomplicated footpath. You see, of late I have found myself making more and more use of the different features of Outlook (something to do with becoming more senior in my company…). Among others, I’ve found that I’ve started using the little flags to mark mails that I have to get back to. But I receive so many mails nowadays that even in a single day the flags get snowed under and pushed off the list. Of course I could just sort by flag status, but I prefer to have today’s mails at the top (rather than the flags and ticks from two weeks ago). So I had myself a bright idea this morning: I’ll just change the grouping of my mails to group by received date and then flag status within the date groups. Walk in the park, right?

Oh mother….

First strike: the very simple idea

Okay, to recap, I wanted to end up with the mails in my inbox grouped first by date of receipt, then by flag status. Now, as it happens, if you do nothing to your Inbox (or any mail folder), Outlook 2007 sorts mails by date by default. You can even see this: just right-click the column headers in Outlook and select Group Box. You’ll see a tree display of grouping fields.

So, simple idea: just drag the Flag Status column into this Group Box to add it as a grouping to the Date. Unfortunately though, Date is an implicit rather than real property that Outlook groups by if there is nothing else and you cannot add any sub grouping to it. You can replace Date with anything, but you cannot add to it. Scratch that idea.

Second strike: a slightly less simple idea

So, the implicit Date as main grouping is out. No biggie, because the Receipt time is included in each mail as a property by Outlook. You even see it as a column in the default layout of every mailbox. Quick job to select that instead of the Date, then subgroup by Flag Status.

The result is, unfortunately, unusable. You see, the Receipt column includes the time as well as the date, even if you format the column to include only the date (the column format is really just for display). Almost no two mails arrive within the same minute. Group on Receipt time and you’ll end up with a million groups consisting of just one mail (maybe two every now and then). So that was that idea up the spout as well.

Third strike: some formula magic

Okay, so we cannot sort on Receipt time because it is too detailed. And there is no date-only column. But Outlook has a neat little party trick: user-defined fields (UDFs). You can define mail items to have custom properties and you can even associate a VB formula with those properties. So here’s our solution: create a custom property which will contain the Receipt time with the time part stripped off. Then use that as the main group selector.

Creating a UDF in Outlook 2007 is easy: right-click the column headers and select Field Chooser. From the drop-down, select “User-defined fields in Inbox”. Then hit the New button and select Formula as type. Fill in a name for the field and in the Formula box fill in

DateValue( [Received] )

Now right-click the column headers again and select “Customize current view”. Hit the “Group by” button. Uncheck the checkbox and select “User-defined fields in Inbox” from the drop-down at the bottom. Now, in the first drop-down for grouping field, select your new field and…. snag. You cannot select the field.

You see, you cannot group by formula UDFs in Outlook. Probably because VB is typeless in principle, so Outlook cannot predict the type of your field and so cannot compare the values reliably. So that idea is out as well.

The long way around

So that means there’s only one way to do this: add a custom property to each mail in the Inbox with the date part of the receipt time. And since it has to be typed field, it has to be added programmatically. So the long road means delving into the guts of VBA for Outlook 2007.

Since we’re going into programming mode here, it is necessary to think a little about what must be done. We want to add a custom date property to each mail in the inbox. But there are two kinds of mails in the inbox:

0. Mails that are already there, which require a one-time loop to add the property.
1. Mails which will come in later and must have the property added as they arrive; this will require event handling.

In both cases we will have to perform the following, general steps:

0. Get hold of the mail M to be augmented.
1. Get M’s receipt time R.
2. Figure out the date part D_R of R.
3. Create a custom property EmailDate for M, with value D_R.
4. Save M with its new property EmailDate.

Visual Basic for Applications in Outlook

Let’s start with some basics about Outlook Basic (ha, ha, I make a funny). Outlook VBA all lives in one large VBA project file called VBAProject.OTM, which contains one project called Project1. Everything has to be added there. Project1 itself consists of a sort of main module called ThisOutlookSession. This is the place to put simple macro’s and the entry points to larger programs.

VBA for Outlook is essentially the same VBA you find in any Microsoft application: a VBA editor (hit Alt+F11 in Outlook) which supports a subset of Visual Basic with some built-in functions and an application-specific object model. In the case of VBA for Outlook this model models things like Folders, MailItems, CalendarItems and so on.

Like all VBA languages, VBA for Outlook allows you to create special variables which can hold object instances that are aware of application events. For example, if you declare a variable like so:

Dim WithEvents inboxItems As Outlook.Items

and then instantiate inboxItems to be the list of items in the inbox, then inboxItems will be aware of any events that occur in the inbox. You can declare event handlers for any interesting events you want to respond to.

Dealing with existing messages

Let’s start simply with the existing messages in the Inbox. To deal with those we need a run-once loop and no event handling. In order to carry out the process outlined above, we will have to get hold of the folder object that represents the inbox in Outlook. We can then loop over the contents of this folder.

While setting the process for existing messages, we will be dealing with mail messages, which are Outlook.MailItem objects. We will be adding a custom property, which is an Outlook.UserProperty object. And we have to deal with the inbox, which is a mail folder — an Outlook.MAPIFolder object.

Getting hold of the Inbox folder is done by getting the default inbox folder from Outlook’s MAPI namespace — this is an object that we can get from the Application object. The Application object is present in every VBA language and represents the application (in this case, Outlook). The MAPI namespace has a getDefaultFolder method which takes a convenient constant to indicate we want the Inbox. After that we can loop over the objects in the Inboxes Items list to set the properties. The result looks like this:

Private Sub SetInitialProperties()
  Dim ns As Outlook.NameSpace
  Dim fld As Outlook.MAPIFolder
  Dim objUnknown As Object
  Dim mail As Outlook.MailItem
  Dim prop As Outlook.UserProperty

  Set ns = Application.GetNamespace("MAPI")
  Set fld = ns.GetDefaultFolder(olFolderInbox)

  For Each objUnknown In fld.Items
    If TypeOf objUnknown Is MailItem Then
      Set mail = objUnknown
      Set prop = mail.UserProperties.Add("EmailDate", olDateTime, True)
      prop.Value = DateValue(mail.ReceivedTime)
      mail.Save
      Set prop = Nothing
      Set mail = Nothing
    End If
  Next

  Set objUnknown = Nothing
  Set fld = Nothing
  Set ns = Nothing
End Sub

As you can see, adding a custom property is pretty straightforward; there are constants to indicate the type (which is why we are using VBA, of course). Getting the date is a matter of manipulating the existing ReceivedTime property of each mail a little. It is worth noting in the loop that there is a check to see if each item is really a MailItem — the Inbox may also contain sub folders, you see.

Next, hit the Run button to run the routine. Then check in Outlook (right-click the column headers, select the Field Chooser and select the UDF’s from the drop-down; drag the new field onto the Inbox columns). The new field contains the date of the mails, but the time midnight (so no more minutes to screw up the grouping). Try grouping by the new column — this time it works, since the new column has a type.

Dealing with incoming mail

In order to set the same property on incoming mail, we need an event handler. As explained above, we can create a special variable that will be made aware of events on any specific object (like the list of items in the Inbox). In fact, that list is particularly useful to monitor since it broadcasts an event every time an item is added. With that information, the actual event handler is pretty straightforward:

Private Sub inboxItems_ItemAdd(ByVal Item As Object)
  Dim prop As Outlook.UserProperty
  Dim mail As Outlook.MailItem

  If TypeOf Item Is MailItem Then
    Set mail = Item
    Set prop = mail.UserProperties.Add("EmailDate", olDateTime, True)
    prop.Value = DateValue(mail.ReceivedTime)
    mail.Save
    Set prop = Nothing
    Set mail = Nothing
  End If
End Sub

Again, note the check for whether an item is a MailItem and not a new sub folder.

As you can see, the VBA convention is to have the name of the event handler be <variable>_<event>(<parameters>). This is also how the VBA interpreter links events to methods (or subroutines in this case).

Now, the event handler is pretty simple. But it does depend on the variable inboxItems being initialized to refer to the list of items in the inbox. So how is this accomplished?

Initializing the inbox variable

Obviously, since Outlook does not provide a database for its VBA macros, there is no such thing as a persistent variable. We will have to redefine the inboxItems variable each time Outlook starts. You guessed it: we need another event handler, written the same way as the earlier one (a subroutine referring to a variable). But isn’t that a chicken-and-the-egg problem? How do we initialize that variable? Luckily, as remarked before, there is one variable which is always implicitly available: the Application variable. And that represents Outlook which, lucky for us, is exactly what we need (we want to listen for application starts of Outlook). So our primal event handler will look like this:

Private Sub Application_Startup()
  Dim ns As Outlook.NameSpace
  Dim fld As Outlook.MAPIFolder

  Set ns = Application.GetNamespace("MAPI")
  Set fld = ns.GetDefaultFolder(olFolderInbox)
  Set inboxItems = fld.Items

  Set ns = Nothing
  Set fld = Nothing
End Sub

Startup is, of course, the event fired when Outlook starts.

Testing the event handler

The simple way to test the event is to run the Application_Startup() subroutine using the Outlook VBA tool to set the system up. Hit Alt-F11, open the ThisOutlookSession module (which contains the code described above), put the cursor in the Application_Startup() routine and hit the Run button. Then go back to Outlook and send yourself a mail. Once it arrives, it will have the new property automatically set.

Just when you think you’re there…

If you’ve been following along, there’s another test you can do (and which I did): close Outlook and start it up again. Then send yourself another mail. If you’re running Outlook 2003 or 2007, the new mail probably will not have the new UDF set.

So what went wrong? We created the event handler, the Outlook startup event handler to set everything up and tested it to see it working? Well, some debugging will tell you almost immediately that the Outlook startup event handler is not called. And the reason, it eventually occurred to me, is macro security. You see, by default Outlook will not run unsigned macros. Even macros defined in its own macro module.

You can get around this easily by going to the security settings for macros (in Outlook 2007 it’s called the TrustCenter) and selecting to run unsigned macros with a warning. But a better idea is to create a self-signed certificate and use that to sign your macro module.

To create the certificate, you need a tool called SelfCert.exe, which is installed with Office in the Office install directory (try c:\program files\Microsoft office\office or c:\program files\Microsoft office\office12). Run the program, fill in a name for your certificate and click OK. Then go to the Outlook VBA editor and select Digital Signature from the Tools menu. Click Choose and select your certificate. Click Ok and then save the module.

Now restart Outlook. It will give you a warning about macros, but if you select “Enable macros” everything will work. You can elect to trust your own certificate in the same dialog, which will eliminate the warning.

Finally…

So, finally, after all that, my Inbox is now grouped by date and then flag status. That took some doing…..

But perhaps someone else will find the experience useful.

Sources

The following forum archive provided me some valuable insights into how to get all of this working: www.eggheadcafe.com/forumarchives/outlook/Oct2005/post24967825.asp

Bad Astronomy

Philip C. Plait. Wiley 2002, Paperback, 288 pages, $8.38

Reading through the book I came to chapter 14, which deals with the doomsday advocates who swore up and down that the world was going to come to an end on May 5, 2000 due to a planetary alignment (more or less) of Mercury, Venus, Mars, Jupiter and Saturn. It was quite funny to read Plait’s description of why this neither would nor did cause catastrophe for our planet. But more than that, the chapter got me thinking about how bad science is all around us and affects us in sometimes rather drastic ways.

In the case of Plait’s example nothing really very drastic happened to anybody (except a few people who were duped into buying bogus survival gear to help them survive the new ice age that the planetary alignment was supposed to drop the planet into). However, not everything that is misunderstood is so innocent. Take for example a court case in the Netherlands (this was quite a long time ago, to be honest):

In this particular case a bridge guard was being sued by a barge captain because the captain felt the bridge guard didn’t open the bridge often enough daily. The bridge was too low for the barge to pass under (by a meter) and the frequent delays cost the captain money. So he sued to get the bridge opened more often. The judge in this case was a long-standing lord of the bench, properly educated in languages and law (as most judges), but not so much in the sciences. His ruling was that the bridge guard could not be forced to alter his ways and that instead, the captain should pay (out of his own pocket) to have the river bed under the bridge dug out an extra meter. It took testimony from a professor of physics of the University of Delft to explain to the judge that that is not how it works.

Lawyers (and related persons) are particularly good at bad science, it seems. And not just physics, but also mathematics and logic. A former professor of mine (a computer scientist, philosopher and master of law) told us some particularly nice anecdotes about the brilliance of Dutch traffic law, for instance. Dutch traffic law specifies that there are priority vehicles (ambulances, police cars, etc.) which have priority in all traffic situations. Dutch traffic law also specifies that traffic signs (signs, lights, arrows painted on the road, etc.) have priority over traffic rules. Makes sense so far, right? Except, my professor pointed out, the law was written by lawyers. Who encoded the two rules mentioned into two separate and completely unrelated articles like so:

A. One must always give precedence to priority vehicles.

B. Traffic signs have priority over traffic rules.

Taken like that, the law says that priority vehicles must stop for red lights. We are all grateful that they disobey the law constantly.

Recently, I got embroiled in an even more far reaching example of bad logic and the law. On some forum somewhere I got into an argument with someone (whom I suspect is a first-year law student) about the Dutch constitution. Our constitution is a pretty typical constitutional document which describes the basic organization of country and government, as well as the civil rights of the citizens and other inhabitants. Perhaps a little unusual (but quite well thought-out, I feel), our constitution recognizes that not every right can or should be absolute. Several of our civil rights are therefore worded in the following structure:

For example, our constitution guarantees the right to free speech; but the constitution also leaves room for the law to set limits (such as forbidding insults to whole groups within society).

The suspected law student, in a bit of brilliant logical derivation, arrived at the conclusion that it is possible for the law in our country to contradict the constitution and therefore that our constitution as a whole is meaningless. That is (to stick with the example above), if it is forbidden to insult all Jews/Christians/Muslims in the country, then the law violates the free speech article and since the constitution is not enforced over the law it means nothing. I’ll admit that the truth is a little subtle: the constitution sets a general rule and allows the law to create exceptions. So the law creates an exception (or violation) of the general rule, but not of the constitution. But still, you’d think that even a law student (a lawyer to be) should be able to spot that he’s claiming that a law whose existence is explicitly allowed by the constitution is a violation of that constitution…

Bad science, I fear, is all around us all the time. Especially since the environment has started playing a major role in decision making. From the greenhouse effect down to whether or not you are poisoning the environment by dumping non-biodegradable plastic in it (you’re not, by the way — non-biodegradable plastic isn’t biodegradable, so it doesn’t interact with the environment if dumped in nature somewhere), bad science pervades life. Two months ago, in The Netherlands, there was a great commotion is some village about plans to store carbon dioxide gas in an old natural gas pit which has been depleted by the national gas seller. Now, this was a porous rock stratum at over a kilometer depth in which nature has stored flammable gas for millions of years, under pressure, without a single leak or explosion. But for some reason the inhabitants (who had been living, technically, on a bomb for centuries before the gas was pumped out) are now scared of living on top of inert, non-flammable gas that is to be stored in the same stratum under less pressure than the gas was under.

Bad science is often funny to read about. But equally as often it is no joke at all, as it leads to bad decisions. When people misunderstand science and especially when bad people are creating bad science in order to make people make bad decisions. So a valid question arises: where are the scientific and engineering communities amidst this storm of bad science? Why don’t they have a more active role in society and why doesn’t society ask them to have a more active part?

In this country, we have several professional societies for scientists and engineers. I’m even a member of one of them (KIvI/NIRIA). Now, it’s not that they don’t try. Really, they do. KIvI/NIRIA, the ONRI, the VSNU, all sorts of others, they all feel the need to play a more active role in society, to influence decision making and to try and be voices of reason amidst the insanity. But they do it in the way engineers have become accustomed to over the past forty years; silently, in the background, trying to be the quiet advisor in the ear of the board. And so they often miss; they get drowned out in the melee, overruled by the opinions of the crowd that are based on nothing that makes any sense. And then, instead of trying to push bask, they submit to the nonsense because the decision has been made and once the decision has been made you no longer disagree. And that may be the worst science of all….