Gridshore » Open Source

Using the NOS open data API with the springframework and jackson

jettro — Wed, 19 Jan 2011 23:36:34 +0000

Some time ago the NOS started providing their data as open data through an API. Using this API you can get the latest news items. You can limit the results to sport news. There is also a search available and a few more categories to find news for. The API is a REST like api and can return xml as well as JSON. In this blog post I am creating an API with the Spring framework RESTTemplate.

First the basics. If you want to try these examples yourself you need to request a key. More information can be found here:

http://open.nos.nl/

Because groovy is a very nice language to do this kind of experimentation I’ll show you de groovy code first. Just a few lines to print a JSON string:

def key = "Your Key"
nos = new RESTClient("http://open.nos.nl/v1/")
JSON newsItems = nos.get(path : "latest/article/key/$key/output/json/category/sport/").data
print newsItems

The result is a long string. If you want to convert a string to nice formatted json check the resources for a link. The following image gives you an idea of the structure of the returned JSON:

Now let us do the same thing in java. Usually this takes much more code. To be honest, it is not at all bad if you use the RESTTemplate of the Spring framework. Have a look at the following code:

public class SearchApiUsingSpring {
    public static void main(String[] args) {
        RestTemplate restTemplate = new RestTemplate();
        String result = restTemplate.getForObject("http://open.nos.nl/v1/latest/article/key/{key}/output/json/category/sport/",
                String.class, "YOUR KEY");
        System.out.println(result);
    }
}

Of course the result is the same, not bad is it? Just three lines of real code. Nice, but not enough for my example. Of course I want to use the result as an object. Therefore I am going to use an object mapper called Jackson. Check resources for a link. Jackson is a framework that enables us to map JSON objects to actual java objects. Based on the structure of the returned JSON I created a java object and used jackson annotations to map it to the JSON structure. The spring RESTTemplate facilitates using mappers. A Jackson mapper is available. The following code shows the improved sample.

public class SearchApiUsingSpring {
    public static void main(String[] args) {
        RestTemplate restTemplate = new RestTemplate();
        LatestArticle result = restTemplate.getForObject("http://open.nos.nl/v1/latest/article/key/{key}/output/json/category/sport/",
                LatestArticle.class, "Your Key");

        for (Article article : result.getItems().get(0)) {
            System.out.printf("Title : %s\n", article.getTitle());
            System.out.printf("Description : %s\n", article.getDescription());
            System.out.printf("Link : %s\n", article.getLink());
            System.out.println("_________________________________");
        }
    }
}

Just a very small difference, besides the printing stuff. The change is the type returned by the getForObject method. Now it is not a string anymore but a LatestArticle. To be able to use Jackson you do need to create the classes with a few annotations.

@JsonIgnoreProperties(ignoreUnknown = true)
public class LatestArticle {
    @JsonProperty("latest_article")
    private ArrayList> items = new ArrayList>();

	// getters and setters
}

@JsonIgnoreProperties(ignoreUnknown = true)
public class Article {
    private String id;
    private String type;
    private String title;
    private String description;
    private String link;

	// getters and setters
}

I use the annotation @JsonIgnoreProperties, I do not want to break my code if a property that is available in json is not available in my java class. I’ll remove it later on when I have all the properties mapped. Why not immediately? Well I want to explain what I did with the date fields first. Mapping a date is a bit different. We do need to map the date using the specific format. I include Joda time to the path and create a synchronizer for a Joda date time object. Jackson has support for joda time out of the box. But I used a different format and therefore I had to create my own. The class Article is slightly enhanced with a DateTime property called published. I only show the setter this time. This is where we have to assign our own deserializer. The code for my deserializer is also shown.

// from Article
    @JsonDeserialize(using = JsonDateDeserializer.class)
    public void setPublished(DateTime published) {
        this.published = published;
    }

public class JsonDateDeserializer extends JsonDeserializer {
    private final DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss");

    @Override
    public DateTime deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException, JsonProcessingException {
        return formatter.parseDateTime(jp.getText());
    }
}

Now we can also print the published date. Be sure not to put annotations on the field and on the setter. I had the problem with the last_update, I need JsonProperty annotation. First I put it on the field, but than the JsonDeserialize was not read anymore.

That is it for now, I will write more when I am ready. Than I will also publish the code. Some of the resources that were useful to me are written down below.

Resources

http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/remoting.html#rest-resttemplate

http://jackson.codehaus.org/

http://jsonviewer.stack.hu/

http://java.dzone.com/articles/how-serialize-javautildate

Book review: OSGi and Apache Felix 3.0

jettro — Wed, 05 Jan 2011 08:59:10 +0000

At the end of 2010 I reviewed the book “OSGi and Apache Felix 3.0 Beginner’s Guide“. The people from packt publishing contacted me based on my involvement in the past with OSGi and felix. I wrote a number of blog posts on OSGi and felix and gave a presentation at the NLJug together with Allard about the topic. During the process of reviewing, I got enthusiastic about the book. I think the result is a good book for people that want to learn about OSGi by getting their hands dirty.

You can find the complete book review on this page: OSGi and Apache Felix 3.0 Beginner’s Guide

Axon Framework 0.6 released

Allard — Sun, 08 Aug 2010 17:53:45 +0000

Today, I released version 0.6 of the Axon Framework. 0.6 has many new features and is another step towards full production readiness. There is still some work to do, but first, let’s take a look at what has changed…

Test fixtures are now part of the axon-test module. These fixtures allow you to test your command handling logic based on events and commands. You define these tests in a given-when-then style: given these past events, when I execute this command, I expect these events to occur.

The CommandBus interface now allows for asynchronous command dispatching. Instead of using a return value, a Callback can be given to the command bus if you wish to be notified of the result of command execution.

The repositories make use of a UnitOfWork instance, which keeps track of changed aggregates and generated events. Instead of saving each aggregate separately, the UnitOfWork can be committed, causing changed aggregates to be stored, and events to be dispatched. With the new SimpleUnitOfWorkInterceptor or the SpringTransactionalInterceptor, it is no longer necessary to call repository.save(). In fact, you are discouraged from doing so.

Repositories are now capable of detecting conflicting modifications. When loading an aggregate, you may pass in the expected version of the aggregate. An exception is raised if that version does not match the actual version.

Event sourcing repositories allow for a more advanced way of detecting conflicts. Sometimes, not all concurrent modification are in conflict with each other. One user may very well change the shipping address of an order, while another user is adding items. Other modifications, however, are clearly conflicting, such as one user marking an order as shipped, and another one adding a product. For this purpose, you can register a ConflictResolver with the repository. This conflict resolver can, based on unseen and newly applied events, decide whether the changes made by the second user are accepted.

Axon now provides a mechanism to trigger snapshot creation based on the number of events needed to load to reconstitute an aggregate. When the number of events exceeds a given threshold, a snapshot is created and stored in the Event Store.

The SimpleCommandBus and SimpleEventBus now expose statistics over JMX. Although currently quite limited, the statistics will be improved in the coming versions.

All of these features help making applications with CQRS-based architectures even easier. But we’re not there yet. The coming months, we will try to make it even easier with features like Event versioning, support for aggregates with multiple entities, and whatever comes to mind in Axon’s growing community.

For more information about the Axon Framework, see www.axonframework.org.

An evening on the Go

Ben — Tue, 27 Jul 2010 23:47:44 +0000

Last Thursday (July 22nd, 2010) Rob Pike, a Principal Engineer at Google, gave a talk at the O’Reilly Open Source conference. In this talk he stated that established languages such as C++ and Java are too complex and not adequately suited for today’s computing environments. He then proceeded with some criticism of dynamically typed languages (that I share) and finally ended up plugging the Go language (which he co-developed) as a solution to the problem.

Now, Rob Pike is not nobody (in addition to being a Google principle engineer he has C and Unix credentials), plus the Go language has the Google brand name on it, so I thought it would be a good idea to check it out….

Evaluating Go

The Go programming language is described on the website as a language geared towards system programming. That is to say, it is a new(ish), general purpose programming language developed internally at Google to scratch an itch some of their developers had and they then decided to open source it.

Evidently, judging from Rob Pike’s talk at O’Reilly, it is also intended to be a simpler alternative to C++ and Java and a more robust alternative to dynamically typed languages such as Ruby, Python and such. It is this claim that I felt was particularly intriguing, so I took a look at the language this evening by reading through the language spec and briefly glancing at the tutorials.

Creak, creak…

Well, let’s address the great big elephant in the room immediately: it would seem that “simpler alternative” is Google-speak (or Pike-speak) for “forty five years old”. It’s had a spit-polish plus language features and libraries added, but there’s no way around it — Go is a rehash of the C programming language with some Pascal thrown in for good measure. Not surprising of course, given Rob’s background, but perhaps not the great linguistic revolution you would have expected.

Just to reassure you all, I’m not exaggerating: Go is essentially C, including structs, pointers and reams of different (un)signed int types. It even includes everyone’s favorite cross-platform language feature: numeric types whose size is architecture-dependent.

Spit-shine

That’s not to say that some improvements have not been made to cover up some of the glaring problems that made C and C++ hard to handle: extern is gone as a concept, as is pointer arithmetic. Plus Go has a type definition feature, so you don’t need macro’s anymore. Also, Go has some more built-in types (such as complex numbers), support for Unicode and it can insert semicolons where needed and has both escaped and unescaped strings (both borrowed from Bourne Shell). And it places the type behind the identifier instead of in front and uses := as an assignment to distinguish from equality (borrowing from Pascal).

Back from the dead

Nevertheless, not all is well with the world. Like it’s ancestors Go still relies on an ability to export named entities (including variables), meaning you can get into the global variable mess. And guess what’s back from the dead: the goto statement, in its full and glorious horror.

So what else is new?

At the core of it, Go is a component-based language. It’s main architectural feature for this is the package, which isn’t much like Java but rather borrows from the Unit found in Turbo Pascal: a collection of exported types and identifiers, with hidden implementations. Packages serve as libraries in the language and are the main strength and source of language functionality.

In the spirit of “this is your grandfather’s technology”, Go is not an object-oriented language. Instead it is fully procedural and uses the abstract data type as its main means of abstraction. Admittedly, this is a good combination with the Unit-like library. The ADT concept has been bolstered with real language support to keep ADT functions and data structures together though. Even though the main data structures are C-style structs, the language includes the concept of a method (which is a function that is sort of legoed into a data type, creating the effect of putting a function pointer into a struct in C). Methods directly associate a callable code block with a data structure, so it’s sort of like adding real behavior to your basic data type. That’s a pretty good concession to OO thinking sure enough, but encapsulation beyond the package level and inheritance are still out.

The other major feature of the language is built-in concurrency modeled on Tony Hoare’s Communicating Sequential Processes. This is a brand of concurrency that is not based around the classic “multi-process, shared single memory” model but around a “multi-process, multi-memory” model whereby processes are linked together by channels. Each process can “toss” values to another process over a channel (or “catch” values coming in across a channel). These operations block until succeeded, which allows for synchronization. The mechanism has been proven equally powerful as Dijkstra’s standard semaphore mechanism. This is once again a typical choice for someone with Rob Pike’s background as this mechanism is also included in the Bourne Shell in the form of named pipes. Of course in such a language it must also be easy to start threads or fork subprocesses and indeed this can be accomplished with the use of a single statement.

So what are we to make of this…

The Go programming language is an interesting combination of proven language features and program architectures. Rooted firmly in the Pascal/C/Bourne Shell era, it seems a bit of a museum of 1960′s language design ideas geared towards the hardware systems environment of that era’s dreams (and possibly today’s reality): heavily multiprocessed with the distinction between local and distributed processing disappeared into the background, using ADT and component-based modularity to achieve separation of concerns.

… except perhaps a bonfire?

And yet, I can’t say that I’m overly impressed with Go. Mostly I don’t see the point. Sure, its well-suited to the multiprocessing environment that we have nowadays. But its support for domain modeling and domain driven design is poor, based as it is on the ADT concept — a serious failing in a modern language if you ask me. And a language that isn’t architecture independent in a Cloud-based platform era? Come on… But most of all I don’t see the new language idea that is supposed to make this language better than all others. I don’t see a refreshing combination of functional and OO paradigms like Scala, or an “easily-typed shell over a large Java class library” like Groovy. I don’t really see what Google thinks it has accomplished with Go.

In addition to that, I don’t see proof for the claim of simplicity. Sure, it covers up the worst pointer-related problems in C/C++. Sure, multithreading is less verbose than it is in Java. But on the whole the language is no simpler than Java, as far as I’m concerned. I don’t like the lacking encapsulation, the poor man’s OSGi modularity, the way methods are tacked on to external data types rather than syntactically part of data structures. And I don’t think first-class lambda functions (popularly called closures) are all that special and I don’t think that having channels as a built-in language element is a great advantage over having them in a separate library (like C++ or Java do).

So what do I think of Go’s future? Well, it certainly has one major weapon in its arsenal: the Google logo, which still makes everything cool that it touches. So Go might get somewhere based on that. But on its own merits I can’t think of any reason to prefer it over another language.

Axon Framework 0.5 released

Allard — Sat, 24 Apr 2010 15:27:59 +0000

Today, I finalized the 0.5 release of the Axon Framework. There is quite a number of changes since the 0.4 version. The 0.5 version is a major step towards production readiness of the framework.

Besides some changes to existing building blocks, such as the event bus, which is now much more powerful, the 0.5 version also includes some new features.

Read on to find out more.

New features

Code restructuring. The package structure has been changed to better reflect the different architectural components Axon Framework provides. It should be easier to find the one you’re looking for.
Command Bus. The command bus is added to Axon. It provides you the ability to explicitly define commands and dispatch them to your command handlers. Furthermore, the command bus provides you the ability to process commands regardless of their type using interceptors. This is useful for, for example, logging, authorization and correlation of incoming commands.
JPA Event Store. The easiest CQRS configuration is on using full-consistency. That means everything should run within a single transaction. Since transactions over multiple data sources involve a huge performance penalty, Axon provides a JPA Event Store. Its performance is not as good as the FileSystem version, but is does provide transaction support.
Easy switching between full-consistency and eventual consistency. You can easily choose to process all commands and related events inside a single transaction, or to handle events asynchronously. Choosing for consistency or high-performance is just a matter of configuration. No coding required.
Per-event listener configuration of asynchronous processing. It is now possible to decide on synchronous vs asynchronous event processing for each event handler individually, just by adding an annotation. If you configure a transaction manager for your event listeners, Axon will process the events in batches and manage the transactions around them.
Support for rolling snapshots. All event stores will automatically pick up snapshot events. Snapshot events are an important performance booster when aggregates generate a lot of events. Instead of reading all passed events, the event store just needs to read the last snapshot event and the regular events created since the snapshot.
Transactional Event Processing. Configuring transactions in asynchronous event processing is now a lot easier. 0.5 includes a SpringTransactionManager you can use in combination with Spring’s PlatformTransactionManager.
Major documentation update. The documentation has been restructured to make it easier to find what you’re looking for.

Maven Central

Where the 0.4 version required configuration of a repository in your project’s pom.xml, the 0.5 version doesn’t. All required artifacts are available in the maven central repository.

Workshop and professional support

We believe that the 0.5 version of Axon Framework is a major step towards production readiness. Therefore, JTeam has decided to provide professional support for the Axon Framework and organize workshops to get you acquainted with the numerous features and choices involved with CQRS.

The first workshop is planned for Friday May 21st in Amsterdam, The Netherlands. For more information, visit http://www.jteam.nl/training/workshop/cqrs-axon-framework-training-workshop.html.

Getting started

Want to get started? Visit www.axonframework.org and download the reference guide. That should contain enough information to get you started. If you still have questions, drop me a message.

Axon Framework – the CQRS framework for Java – version 0.4 released

Allard — Sun, 21 Feb 2010 15:57:56 +0000

Last week, I published the 0.4 release of the Axon Framework. Axon helps developers build high performance, scalable and extensible applications using the CQRS pattern. The 0.4 release is a major step towards 1.0, and includes transactional event handling, high-performance caching repositories and easy configuration of event sourcing support. Furthermore, we have also built a demo application that uses Flex to get real-time updates pushed from the server.

Read on to find out more.

About the Axon Framework

Axon Framework helps build scalable, extensible and maintainable applications by supporting developers apply the Command Query Responsibility Segregation (CQRS) architectural pattern. It does so by providing implementations, sometimes complete, sometimes abstract, of the most important building blocks, such as aggregates, repositories and event the bus – the dispatching mechanism for events. Furthermore, Axon provides annotation support, which allows you to build aggregates and event listeners without tying your code to Axon specific logic. This allows you to focus on your business logic, instead of the plumbing, and helps you makes your code easier to test in isolation.

Features available in 0.4

We have made quite a few additions and changes since the 0.3 release. Unfortunately, this also means I had to make some minor API changes. There is now a bigger choice of abstract classes for most of the building blocks. This allows you to choose whether you want to use the logic provided by Axon, or prefer to develop more of your own. Whatever your wish is, there should be a class to help you on your way.

The biggest change since 0.3 is the addition of an Asynchronous Event Bus that supports transactional processing of events. Since it is asynchronous, it means that your command processing doesn’t have to wait for events to be processed. However, this Event Buss can also guarantee the order in which events are delivered to the event handlers. You can choose full concurrent processing (without any guarantees about ordering), full sequential processing (FiFo guarantee) or anything in between. You could, for example, guarantee the sequential processing of Events that come from the same aggregate.

The Event Bus also provides transaction support. That means you can tell the event bus to retry an event after a certain amount of time when processing fails, for example due to an error in the underlying mechanism, such as a database. For performance reasons, you can configure the event handler to process multiple events within a single transaction. Database updates are a lot faster if you process several events within a single transaction before committing it. Within your event handler, you can configure transactions.

Besides Domain Events, Axon now also allows you to explicitly define two other types of events: Application Events and System Events. Application Event can notify your event listeners that something interesting happened in your application. System Events are a special type of application events and can be used to indicate that a part of your application has changed state, for example when the database is no longer available. These System Events could provide interesting operational information about your application.

Another major change since 0.3 is the documentation. We have brought the documentation up to par with the code-base. All features are now extensively described in the reference guide. You can download the manual from the Axon Framework website: www.axonframework.org.

Last, but definitely not least, we have created a demo application that you can easily download and deploy to have a look at how you can use events to your advantage. This demo application is built using a Flex Client and an Axon-based server side component. The Flex Client registers itself as an event listener and will automatically process incoming events, at real time! That means that you can have 2 windows open, change something in one window, and see the change come in immediately on the other window, too. Check our Sample Installation page for more information.

What to expect in upcoming releases

The major components of the CQRS pattern are already available in the 0.4 release of Axon Framework. However, there is still a number of features we want to include in the 1.0 release. Although one of the advantages of CQRS is that it makes an application scalable and extensible, the 1.0 release of Axon Framework focuses on what happens inside a single JVM. Streaming events between JVM’s is not made impossible though, the Spring Integration connectors can be used to publish events to JMS, MQ, mail, HTTP and any other protocol supported by Spring Integration.

Some features to expect before the 1.0 release are rolling snapshots. When aggregates live for a long time, the number of events to read in each time the aggregate is loaded could easily run in the thousands. Reading in a thousand events takes a long time. Instead of reading in a thousand events, you can summarize all these events into a single snapshot event. Axon Framework will provide the plumbing necessary for you to build snapshot events.

Another feature that we’ll be spending some attention on is failure recovery. Unfortunately, applications sometimes behave unexpectedly and sometimes stop entirely. The reason of a crash won’t be the Axon Framework itself, of course. But since we have a reliable source of events in the events store, we can use this to recover from failure. Axon Framework will provide some building blocks that allow you to republish events to event handlers that did not get the chance to process them. This same mechanism will allow you to restore your application state from backups.

The last feature currently on the roadmap is JMX support. Since Axon Framework deals with the dispatching and invocation of event handlers, it has interesting information about an application’s performance and technical state. We are planning to expose this information using JMX MBeans.

If you have any other ideas, you can submit a feature request on www.axonframework.org/issues.

Getting started with Axon

If you’re eager to get started using Axon Framework for your own application, check out the Axon Framework website. There, you can download the binaries, sources documentation and a sample application. If you use Maven, the reference manual will explain how you can configure the dependencies on the Axon binaries.

If the information on the website is not enough to get you started, don’t hesitate to send me a message or post a comment at the bottom of this blog entry.

A special thanks to…

Before I wrap up, I would like to thank Jettro Coenradie for his help on the sample project and the reference documentation. I think the sample turned out into a pretty cool demo environment for some of the advantages that Axon can provide.

Also many thanks to JTeam for the time they allowed me to spend on the project. Without it, I am sure I couldn’t have delivered this release. Thanks!

CQRS made easy with cqrs4j

Allard — Mon, 21 Dec 2009 08:37:00 +0000

Command Query Responsibility Segregation (CQRS) is an architectural style that makes a clear distinction between commands, which tell an application to do something, and queries, which are requests for information from an application. This distinction comes from the fact that the requirements (and thus also the model) for the execution and validation of commands are significantly different than those for queries. Events play an important role in the synchronization of application state resulting from executed command.

Applying a CQRS style architecture involves the development of quite a lot of “plumbing” code: event dispatching, asynchronous event processing, transactions, etc. cqrs4j, an Apache 2 licensed open source framework, takes care of all the plumbing for you. Read on to find out how…

A brief introduction to CQRS

As I stated above, CQRS makes a distinction between the model that validates and executes commands and the model that is used for providing state information. This is quite significantly different than how most (web)app applications are build nowadays. In fact, the model used for commands (the box named “Domain” in the image below), does not expose any state, at all. You might wonder, but how do I know what to show in the front end? That comes directly from your data sources using a thin data layer. I’ll explain how the data gets there in a minute.

When a command comes in, it will load an aggregate from the repository and execute certain operations on it. As a result of these operations, the aggregate produces events, which are picked up for storage by the repository and for dispatching by the event bus. The event bus will dispatch each event to all (interested) event handlers. Some of these event handlers will perform actions on other (related) aggregates, some others will update the database tables they manage.

Having handlers update the data in the database means that your tables do not have to be normalized anymore. Instead, CQRS allows you to optimize your tables for the way you want to query them. This makes the data layer processing your queries really simple, and maintainable.

Furthermore, since all state changes are initiated by events, these events become a reliable source for an audit trail. By storing these events, you have an audit trail that you can use to replay the entire application history. Instead of just seeing “current application state” only, you can see all the changes that have led to the current state, providing valuable information trying to find bugs or deal with customer complaints.

The asynchronous and event driven nature of CQRS makes it extremely valuable for complex applications that have different views on the information in the application. Integration with third party systems that need to be notified of certain changes in your application –and vice versa– is a lot easier if you use an event driven approach. And since event handling is done asynchronously, the application is more responsive and easier to scale.

But as you notice, there is quite a lot of processing and pushing around of events. When you have a customer breathing up your neck, you don’t want to be developing infrastructure code. You would want to be focusing on the business logic, which is quite complex by itself.

cqrs4j

cqrs4j is a Spring-oriented framework that provides the most important building blocks of a CQRS architecture. Creating an aggregate becomes really simple. You don’t have to worry about managing the storage and dispatching of (uncommitted) events. Event sourcing becomes as easy as wiring two beans in your application context.

Annotation support

cqrs4j comes with out-of-the-box annotation support. This makes is really easy to wire event handler methods. cqrs4j can automatically subscribe all your event handlers to the event bus and delegate all relevant events to the appropriate event handlers. Enabling annotation support is as easy as wiring a single bean in your application context.

Spring Integration support

Spring Integration is a framework that allows easy developments of a Messaging systems using the pipes-and-filters architecture. This fits nicely with most of the event dispatching process of CQRS. cqrs4j has support for Spring Integration, which allows you to publish all events as messages on a Spring Integration channel. Sending messages through JMS queues, via email, or through file system storage will only take a few lines of (XML) configuration.

Transaction support

If you update your database tables through incoming events, dealing with them one-by-one in a transaction can become time consuming and take up too much resources of your database. With cqrs4j, you can configure transactions just by setting the @Transactional annotation on your event handlers. You can also configure how many events should be handled in a single transaction.

cqrs4j project page

cqrs4j is an open source project, licensed under the Apache 2 license. Visit the project home page for the latest downloads, documentation and source code at code.google.com/p/cqrs4j.

We welcome your feedback

If you have any requests, remarks or other feedback, please let us know. You can report bugs and improvement requests on the issues page. You can also leave a message via the contact page or by leaving a comment.

Open source as in free

jettro — Fri, 14 Nov 2008 20:34:32 +0000

Last week I was at the NLJug J-Fall. This is a conference for Java developers organized in the name of the Dutch Java Users Group. One of the keynotes was given by Reginald Hutcherson from Sun. He talked about Sun and of course also about making Java and many related products open source. He explained why this was very good. Since we all love free software. This was obviously a comment with a humorous undertone. Yet there are still many people who truly believe open source is just free software. What is really free in this context? Like many others I am think that open source software is not free.

Read on to learn why

When we look at the overall picture of software, there are a number of levels of free. In my view we can distinguish the next 4 steps in “free” for open source:

Free source code
Free software
Free documentation
Free Support

How free is an open source product? You can do with the sources what you want? For example, create your own components and sell your components together with the open source product? To explain what you can do with the sources, an open source products is accompanied by a license. Some licenses feel like a free newspaper on the street.

“Sir, you want to try this newspaper for free?-Yeah, sure, give it to me-but you’ll need to register here for a trial subscription that without written termination is immediately converted into a permanent subscription.”

So a free paper comes with a subscription and some rules around this subscription. Like with this free newspaper, free open source software also comes with a license. The instructions might indicate that there are multiple licenses depending on your type of use. Important is that some licenses do not go together. Therefore I think free software becomes quite complex.

The more free an open source product is, the more important the community behind the open source product is. If only the source code is freely available, you’ll need to build the product yourself. Besides the source code is perhaps an installation manual available. In this situation the risk of problems is large and so the support of a community is very important. This support will focus largely be confined to a mailing list or an online forum. Fortunately, the most open source projects also provide “runnables”. In other words, working applications that can be executed immediately. If you’re lucky, you can also find free a user manual. Support is available via mailing lists and online forum.

The next step is free documentation. Again, a very active community is important. Without an active community there is often no proper documentation. Moreover, proper documentation results in better overall adoption of a product. Another reason for good documentation is a commercial interest for a company. The most common form of a commercial interest in an open source project, is a company that arises from the starter (s) that open source project. This company will assist in the use of the open source product. Now it’s interesting, especially as the last step in the free software is free support. This undermines the business model of the commercial business, since they often earn money by providing support. The balance between free and paid support is therefore difficult for a company that wants to make money and not loose the community. Therefore, online support is usually free. This is picked up by the community together with employees of the company that supports he open source project.

Free means that you can obtain the sources for free, not necessarily the use of the software. You still need someone who goes into the sources, learning about the software or someone who will understand the software so as to train others. You probably need to train people in using the software before they can become productive.

I believe that open source will never be really free. What does this mean for open source now? A disaster? Nope, with paid software it is no different. Then you must also train your people. Open source is not the same as free. If you take the words open source literally, it means that only that the source code is open or available. The “Open Source Initiative” has a definition that describes open source. The availability of the source is only one of the 10 points which open source must meet. The other points that are described, among other open source software should be able to pass on to others, provided that the license is included, the original writer gets recognition. Read the full 10 points on the following page: http://www.opensource.org/docs/osd. A license may call itself an open-source license only if it is approved by OSI. A long list of open-source licenses is available on the same website: http://www.opensource.org/licenses/alphabetical. When you have read the open source definition, you come to the conclusion that there is no talk of free software. There is only talk about the availability of open or free sources and what you can do with those sources.

Conclusion, open source projects tend to make their software available for free under a well known open source license. The software can be used for for free. You can ask for help for free in a forum. For onsite support you often have to pay. Because you can do with so much for free with open source software, open source will often be seen as free. I think however that the use of open source will never be completely free. You’ll always have to invest in learning about the software. You may have to hire expensive consultants for implementations. The main lesson learned that I can give is that making a choice for open source should never be based on the fact that it is “free”, this is not right. Opt for open source for the freedom that you get, the quality and particularly the community that is always willing to help.

Yes We Can….the day after

Ben — Wed, 05 Nov 2008 16:35:22 +0000

On November 4, 2008 a great wish came true for most of the United States and the rest of the world: a great alliance of liberals, progressives, new voters, young voters, men, women, blacks, hispanics and people who wanted change in general rose up and dropkicked George W. Bush and his neoconservative cronies out of power in Washington D.C.

So now it is November 5. It’s the morning after and instead of being just the junior senator from Illinois, Barack Obama is now the president-elect of the United States. Which means that the time has finally come for those of us who cannot vote to start thinking about what we can expect in the way of change. Of course we all know about the big things that have been central themes in the campaign, like the economy and the war in Iraq. But for us who work in the technology sector it is also interesting to wonder what changes might be on store.

Now, I am not a real political pundit. So I might be talking out of my ass in the rest of this article. But nevertheless what follows is, in no particular order, the topics that I think will be interesting to watch in the next few years in the United States under new management.

Energy as a technology driver

The Obama campaign has never had much to say on the issue of technology per se (it’s not foremost an anybodies mind right now, so that’s understandable). Where Obama has touched on technology, it has almost solely related to energy technologies (renewable energy, clean coal technology and so on). That isn’t much numerically, but as a topic it is certainly a very good one to commit to pushing technology.

Energy technologies are currently the bleeding edge of applied physics and chemistry, with an enormous push in technology and knowledge development in order to create energy sources that do not deplete the world’s resources and limit carbon dioxide emissions. They are also poised to become a huge industry driver with all sorts of new plants, facilities and infrastructure going up all over the place. Which offers opportunities for all sorts of companies and industries from all over the world. Whether it is Siemens, General Electric or a startup like Darwind (producers of wind turbines), producers of tidal turbines or solar panels, solar power stations, or large energy consumers (like the Google data centers or the Amsterdam Internet Exchange or greenhouses and chemical production plants) looking to reduce their energy footprint, or technology providers (companies, universities, research institutes) developing technologies to help others, there is enormous potential everywhere.

And, while it may not seem so at first, part of that potential is also in smart IT solutions. I’ve already mentioned power usage by Google and others. Energy conservation is one answer, but part of the solution will also have to come from incorporating new energy sources into the infrastructure — solar, wind and water. Some of the new sources may be small (like energy production from solar panels on people’s houses) and many will suffer from high fluctuations in supply. In order to accommodate this new paradigm power grids will have to be replaced by or upgraded to smart power grids (strongly dependent on computer control systems) that can dynamically switch on small-to-large supply fluctuations, can rebalance the system in these conditions and can allow for correct billing so that home power producers can be paid for the power they put into the grid. Google and General Electric have already signed an agreement to develop such grids together in the future.

And other examples are rife. Lots of machinery and machine networks will start using dynamic controls to optimize power use and minimize emissions. One of the major challenges here (since it is such an open system with many small parts that cannot all be replaced or upgraded at once) is the traffic infrastructure. The major attention here is on hybrid engines, hydrogen engines and soon. But the fact is that huge inroads will be made by having more automation in cars to balance power distribution to the wheels, break control and balance, traffic jam avoidance and (in the future) guidance control.

As you can see there are huge areas where energy concerns can be a technology driver in the upcoming years. It will be interesting to see whether and to what extent the new Obama administration will push technology development in this area in the next few years. Will Obama be able to make a large energy conservation and renewable energy push in the U.S.? Will he put people in charge with the vision to see how far-reaching the new developments can be? And how will Europe and the rest of the world respond? Will they be spurred on to even greater heights in this field, inspired by a new U.S. push and driven by competition?

The defense industry

Aside from new fields, such as energy, the defense industry has been a technology driver in the U.S. for the last five decades. This area will be interesting to watch as well.

On the one hand, Obama is faced with a rotten economy and impopular wars. That is not the climate in which to invest heavily in the technologies of war. Rather, I wouldn’t be surprised to see defense budget cuts coming up in the next few years. On the other hand, DARPA has always done pretty well for the U.S. and I cannot imagine any president seriously cutting back on that agency.

What will be most interesting to see is if the Obama administration will refocus the defense industry to move from shock-and-awe baloney to a smarter, leaner way of working. DARPA has long funded research into UAVs all over the world (as have other governments and institutes). And (to look elsewhere shortly) the Dutch navy has been developing naval vessels for years that can be manned by smaller and smaller crews. Will the Obama administration move to a more high-tech and less bulky military? It would be interesting, especially since such a military would depend heavily on IT development as well.

Education in general

Obama made a big deal of education in his campaign. And rightly so, since primary and secondary education in the United States is an ungodly mess. But aside from pure necessity to keep the United States from sliding backward in the future, a major education push may have profound long-term effects on the IT industry.

One of the major reasons for the popularity of outsourcing to India in the last few years has been the ready availability of a large pool of at least reasonably well-educated software developers. And their low cost of course, although that effect is melting away rapidly. But cheap or not, Indians do still have the advantage of being available, whereas in Europe and the U.S. people are still hard to find. But an education push might just change that in the next generation. If more people reach higher levels of education the U.S. might once again be able to fill a large part of its own requirements. In combination with the rising costs of Indian and Chinese resources (already a reason for Intel to move some of its production plants back to the U.S.) we may see a small or even dramatic reversal in the outsourcing trend in the next 15 to 25 years.

Something else might prompt a small, short-term reversal as well by the way: the new commitment from the Obama administration to make it harder and costlier to ship jobs overseas. That in combination with economic slowdown in the next year may very well cause a small collapse in the outsourcing trend from the United States if the Obama administration is willing not only to make overseas jobs more expensive but to start subsidizing local jobs to make them more attractive to companies. Which of course opens the question of what will happen if Europe follows suit.

The Microsoft case

This one is a bit of a sideshow nowadays, but still: Microsoft might suddenly find life a little harder again under the Obama administration. In the latter days of the Clinton administration, Janet Reno was prosecuting Microsoft for antitrust violations. Microsoft heavily backed the Bush campaign in 2000 and in 2001 the case against them suddenly vanished. Now the Democrats are resuming executive control, Microsoft might start feeling the crosshairs of the Justice Department on its forehead again.

Of course there is the question of whether this still matters. Microsoft has not been doing all that well in recent years and the mantle of technology leader company seems to have passed to Google more or less. Still, it’s something to watch.

The Internet Generation

The final thing that I think we should watch for in the next few years is less tangible and therefore the most interesting: the effect of the Internet generation. Obama was elected in part due to heavy support from young adults in the United States and it shows in all aspects of his campaign. From the fact that he is the first president(-elect) ever who gave the online portion of his campaign equal footing with the real-world part to the slick use of all media to appeal to voters… It is clear in everything, this will be the first administration populated heavily by people from a generation that has grown up in an online world.

The question now is what effect in general this will have on the relationship of the administration to technology, technology users and their views on the world. I’m not exactly sure where this could go, but I think two interesting topics to watch are the new administration’s views on intellectual property law and its views on open source and government.

The second subject is of direct interest to the technology field. There have been half-hearted moves in the last few years in different countries towards using open source software rather than commercial software (and one serious move in Brazil). Perhaps the United States will adopt a similar policy, or at least make some moves in that area in some parts of the government. This might also be interesting in combination of the passing of the mantle of leadership (or “coolness”) from Microsoft to Google and might be fuelled by the rise of the Google/Internet generation (after the Microsoft/Desktop generation). Either way, there might be reason for a far greater push into OSS than we have seen even in the last few years.

The first subject might be the result of the Internet generation’s new views on information exchange and availability of everything you want when you want it. The world has already known for years that a new way of dealing with intellectual property (especially copyright) is necessary in the new world. It will be interesting to see how much influence the youth component will have on the voice of the United States in the next few years.

Waiting for the future

Like I said, I think there are lots of interesting developments that are coming our way courtesy of the new management in the United States; I hope, after reading, that you agree. We already have the beginnings of change across the pond; all that remains now is to see what shape change will take. And how we will respond to change to make life better in our jobs, our lives and the lives of the people around us.

All of a sudden, life is exciting again.

YES, WE CAN!!!

Redesigning libraries completely

Ben — Wed, 15 Oct 2008 22:53:23 +0000

I just got done reading Jettro’s post on refactoring your library. And still being in the vacation mood, it reminded me of an idea that surfaces in my so-called brain every now and then: the public library for the twenty-first century.

Much like Jettro I have the habit of accumulating odds and ends of documentation over time (sometimes including entire books which tend to go out of date). Which means that I tend to run out of space after a while. At the same time all this documentation suffers from the problem of not being very portable, which means it doesn’t fit very well with our information at your fingertips age. All in all a digital form of storage available at any time and possibly to a large audience (or at least available to me from any place) would be far preferable. However, that solution runs afoul of copyright laws and treaties.

Looking at it in a more general way, there is a general contradiction between the historical rights of authors and the needs of information consumers in the modern age. In the following sections I will attempt to provide an overview of the reasons and interests in this situation and what I think would be a good idea for our modern age.

Copyright

Why copyright?

Let’s start right at the beginning by reviewing why we have copyright in the first place.

Copyright started in fifteenth century England. Until the fourteenth century, if you wrote something your only way of keeping your creation safe was by obscurity: show it to nobody, let nobody read it or hear you reading it. Now, for most of history that was not a great big problem; most people didn’t read, very few could write and what they did write they actually wanted heard (plays and stories) or was too complicated for most people to understand (early works of science). It was the time of the storyteller, the playwrite and the scribe (and in fact quite a large number of works from earlier centuries were lost due to lack of dissemination).

But in 1444 Johannes Gutenberg invented the movable type printing press and for the first time made it possible on a large scale to record and disseminate ideas, theories, stories, images and far more. For the first time ever what one man thought of might be carried across cities, countries and continents at a speed far beyond his control. And for the first time it became possible to steal ideas for ones own profit.

At first of course this was not a major problem. The only work of mass popularity available was the bible and nobody was really claiming ownership of that (although its large-scale dissemination would have huge repercussions a century later). But as other works spread, science started taking hold in southern Europe and wealth started to accumulate in the seafaring kingdoms of Europe, it became necessary to protect works from which one might actually derive profit. Which of course severely limited the dissemination of knowledge and so stood in the way of further development.

As progress took hold over centuries and Europe started to be a place for knowledge-capital again, it was becoming necessary to introduce a system whereby a person could make his works public without thereby giving up all control over that work an lose the possibility of deriving personal gain and recognition from it.

What is copyright exactly?

Copyright was the answer to this quandary and is more or less exactly what it sounds like: the right to copy a work. More specifically, copyright is defined as follows:

Copyright: The monopolistic right of the author of a work of science, art or literature to

publish and/or

disseminate

that work.

Copyright is a legal mechanism that allows an author of a work to make his work public in any way that he desires for his own personal gain and have government support to keep everybody else from stealing that work after it has been made public. This mechanism benefits the author in that he can profit from his work and still keep control of it. And at the same time it balances the needs of the author against the needs of knowledge consumers who can now access the work in question once they’ve gotten permission from the author, pass on their copy of the work (but only that copy) and quote or otherwise use the work as long as they give credit to the original author.

Copyright is a form of legal protection for authors which is given by national governments. As such, copyright is a matter of national law in every country. However, since most works are spread across borders nowadays (and have been for a while) there is also an international treaty that creates a protective framework for authors across most of the globe. This treaty is known as the Berne Convention and is governed by the World Intellectual Property Organization.

A modern age of publishing

Publishing and the rise of the Internet

The rise of copyright and the system of laws and treaties establishing copyright (especially since 1886) were a very reasonable and acceptable compromise between the needs of authors and knowledge consumers for a long time. The system allowed for the controlled but ever-growing flow of information, which in turn allowed more and more people access to all sorts of knowledge and allowed them to produce, record and disseminate more knowledge themselves and so on. It is fair to say that the technological revolution that has taken place on this planet starting with the Industrial Revolution would not have taken place without copyright.

However, in the 1960′s the continuous growth of knowledge of the last centuries resulted in a dramatic change of circumstances for authors of all works; the kind of change that had last been seen in the 1440′s. On October 29, 1969 two teams working for DARPA created the first part of the Internet and in so doing (although I’m sure nobody stopped to think about it at the time) created a whole new medium for disseminating works. A medium that, much like the movable type of five hundred years earlier, vastly increased the speed and ease with which any work might be disseminated amongst a huge number of people and which might also enable those people to understand those works far more than before.

Copyright worked well for a long, long time because it was a mechanism suited perfectly to the media available when it was invented. Works were available and could be disseminated under copyright from a relatively limited number of sources to a large audience which was, on the whole, capable of receiving but not of reproducing works (at least not in a way that would seriously threaten the livelihood of the authors). Also, the limited number of original sources of dissemination made it possible to institute a reasonable system of compensation for authors of works which were subject to mass dissemination (like songs broadcast on the radio). The introduction of the Internet made it possible for the first time for every dissemination of a work to result in a new source of dissemination with ease and speed. Copyright, although not encoded to be limited to a particular collection of media, was not prepared for that.

Open source, open standards, open everything

Of course, as usual, nobody noticed at first. For starters it took a while for the Internet to become a popular medium; that didn’t happen until the latter part of the 1990′s. Second, the dramatic impact of the Internet on traditional views on copyrightable material also required another development: the popularization of the idea of information freedom.

Freedom of information (i.e. the idea that information should be freely disseminated) started in earnest in the 1960′s as well at universities such as Berkeley, where the first generation of students to “grow up” with computers and networking saw the potential and ease of digital media in distributing data. This new idea took hold in many forms and led (among others) to the development of open source groups like the GNU, whose philosophy is that of free software.

The result of these developments was that the next generation grew up in a new, digitalized age with a whole new attitude towards works and intellectual property. Copyright had always been an understandable and reasonable concept because it fit the available media and it was hard to seriously violate copyright laws without trying (you cannot accidentally copy a whole book fifteen times). But in the new age, where copying is easy, copyright seems a lot less reasonable and a far greater limitation on the will of people who are used to easy access to everything. Which is a problem, since one cannot deny that the more freely information flows, the more and more varied use it finds and the more and more varied growth of knowledge results.

Supranationalism and the level, European playing field

There is one more recent change that I must mention that is relevant to what I want to propose. After the Second World War, the countries of Europe (and to some extent the world) realized that more and more often the world is faced with problems and challenges that surpass the traditional, national boundaries — and the abilities of nation states to deal with. For more and more things it is necessary to have supranational cooperation and a system of rules and regulations that don’t stop at the border.

Luckily, in Europe, we have something like that: we have a European Union. The European Union is a supranational organization dedicated to pursuing ever closer union among the nations of Europe. And part of that mission is to strive towards an ever more level playing field for people and businesses in all walks of life. To create a society on top of different societies with both cooperation and fair competition across traditional boundaries.

That same European Union is also charged with the task of creating a situation of ever-increasing opportunities on equal footing for all its citizens in this modern age. As such, the Union also faces the apparent clash between copyright and the need to disseminate information to stimulate development of the European society through science, industry, culture and education.

The European library

So, what do I think should be the answer? Where do I think the new balance should be struck? Well, I think that there is a huge opportunity right at our feet if we only dare see the circumstances of the day as a chance and not a problem. To recap the above, we are faced with the following:

A body (by necessity ever growing) of authors in need of protection of their rights
A body of knowledge consumers empowered by a new medium no longer to be constricted by the old ways of doing things and not feeling those old restrictions to be reasonable anymore
A new form of government, encompassing all the old and unifying many different people, rising on the need for cooperation between peoples

Now, call me crazy, but it seems to me that we have not only a problem, but the necessity and the means to solve that problem and a perfectly positioned group of people to take responsibility for implementing the solution.

Back when mass media was first introduced, copyright law faced similar problems as it does nowadays but on a more limited scale. The solution back then was to allow mass media dissemination of works and arrange for compensation at the source. We have the same situation today except that we have no organized source from which to arrange compensation, nor a large enough government institution to arrange compensation.

So what I propose is this: since the desire for dissemination of works exists among all the peoples of Europe, let it be a task of the European Union. Let it not fall to the Union to fight the tide by trying to enforce ancient copyright, but instead let it fall to the Union to arrange dissemination of all works to all European citizens through an online library offering every European citizen access to everything — books, music, movies, scientific papers, everything else, old, new, everything ever published in Europe before, now and in the future. And let it be a service that is paid for by the European citizens through common taxation — carried by all Europeans, free at point of use and with as much use as wanted or needed. We have the technology required (the Internet), the government required (the Union), the driving forces (the authors and the knowledge consumers). All that is required is the will to embrace a new way of thinking and the courage to let go of the old.

Now, just in case you think I’ve gone off my nut: I have no illusions that this is going to happen (or at least not anytime soon). But it is what I would like to have happen and what I think should happen.

Modern reading equipment

All of that aside, there’s another thing I would like to have happen to go with that European library: a big push in the electronic paper area. Or at least a vast improvement in Internet connectivity and devices, because I really dislike everything we use to gather information online nowadays. Both desktops and laptops are incredibly cumbersome to handle and force you to sit (or lie) in uncomfortable positions for far too long simply because you can’t make them move with your body and so must make your body accommodate these devices instead. As I already said, I don’t imagine we’ll have a European library anytime soon — so can’t we at least have devices that aren’t trying to cripple us instead?

[0] – Treaty on the European Union, Title 1, Article 1, paragraph 2