Gridshore » Spring Framework

Using the NOS open data API with the springframework and jackson

jettro — Wed, 19 Jan 2011 23:36:34 +0000

Some time ago the NOS started providing their data as open data through an API. Using this API you can get the latest news items. You can limit the results to sport news. There is also a search available and a few more categories to find news for. The API is a REST like api and can return xml as well as JSON. In this blog post I am creating an API with the Spring framework RESTTemplate.

First the basics. If you want to try these examples yourself you need to request a key. More information can be found here:

http://open.nos.nl/

Because groovy is a very nice language to do this kind of experimentation I’ll show you de groovy code first. Just a few lines to print a JSON string:

def key = "Your Key"
nos = new RESTClient("http://open.nos.nl/v1/")
JSON newsItems = nos.get(path : "latest/article/key/$key/output/json/category/sport/").data
print newsItems

The result is a long string. If you want to convert a string to nice formatted json check the resources for a link. The following image gives you an idea of the structure of the returned JSON:

Now let us do the same thing in java. Usually this takes much more code. To be honest, it is not at all bad if you use the RESTTemplate of the Spring framework. Have a look at the following code:

public class SearchApiUsingSpring {
    public static void main(String[] args) {
        RestTemplate restTemplate = new RestTemplate();
        String result = restTemplate.getForObject("http://open.nos.nl/v1/latest/article/key/{key}/output/json/category/sport/",
                String.class, "YOUR KEY");
        System.out.println(result);
    }
}

Of course the result is the same, not bad is it? Just three lines of real code. Nice, but not enough for my example. Of course I want to use the result as an object. Therefore I am going to use an object mapper called Jackson. Check resources for a link. Jackson is a framework that enables us to map JSON objects to actual java objects. Based on the structure of the returned JSON I created a java object and used jackson annotations to map it to the JSON structure. The spring RESTTemplate facilitates using mappers. A Jackson mapper is available. The following code shows the improved sample.

public class SearchApiUsingSpring {
    public static void main(String[] args) {
        RestTemplate restTemplate = new RestTemplate();
        LatestArticle result = restTemplate.getForObject("http://open.nos.nl/v1/latest/article/key/{key}/output/json/category/sport/",
                LatestArticle.class, "Your Key");

        for (Article article : result.getItems().get(0)) {
            System.out.printf("Title : %s\n", article.getTitle());
            System.out.printf("Description : %s\n", article.getDescription());
            System.out.printf("Link : %s\n", article.getLink());
            System.out.println("_________________________________");
        }
    }
}

Just a very small difference, besides the printing stuff. The change is the type returned by the getForObject method. Now it is not a string anymore but a LatestArticle. To be able to use Jackson you do need to create the classes with a few annotations.

@JsonIgnoreProperties(ignoreUnknown = true)
public class LatestArticle {
    @JsonProperty("latest_article")
    private ArrayList> items = new ArrayList>();

	// getters and setters
}

@JsonIgnoreProperties(ignoreUnknown = true)
public class Article {
    private String id;
    private String type;
    private String title;
    private String description;
    private String link;

	// getters and setters
}

I use the annotation @JsonIgnoreProperties, I do not want to break my code if a property that is available in json is not available in my java class. I’ll remove it later on when I have all the properties mapped. Why not immediately? Well I want to explain what I did with the date fields first. Mapping a date is a bit different. We do need to map the date using the specific format. I include Joda time to the path and create a synchronizer for a Joda date time object. Jackson has support for joda time out of the box. But I used a different format and therefore I had to create my own. The class Article is slightly enhanced with a DateTime property called published. I only show the setter this time. This is where we have to assign our own deserializer. The code for my deserializer is also shown.

// from Article
    @JsonDeserialize(using = JsonDateDeserializer.class)
    public void setPublished(DateTime published) {
        this.published = published;
    }

public class JsonDateDeserializer extends JsonDeserializer {
    private final DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss");

    @Override
    public DateTime deserialize(JsonParser jp, DeserializationContext ctxt) throws IOException, JsonProcessingException {
        return formatter.parseDateTime(jp.getText());
    }
}

Now we can also print the published date. Be sure not to put annotations on the field and on the setter. I had the problem with the last_update, I need JsonProperty annotation. First I put it on the field, but than the JsonDeserialize was not read anymore.

That is it for now, I will write more when I am ready. Than I will also publish the code. Some of the resources that were useful to me are written down below.

Resources

http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/html/remoting.html#rest-resttemplate

http://jackson.codehaus.org/

http://jsonviewer.stack.hu/

http://java.dzone.com/articles/how-serialize-javautildate

Playing with the spring mobile project

jettro — Sun, 28 Nov 2010 20:02:13 +0000

Spring seems to be all over the place today. Interesting projects just seem to keep coming out of their development machines. Hard to keep up. In my previous blog I had a look at the spring-social project. In this blog post I’ll have a go at the spring-mobile project. I’ll go through the things you have to do for configuration and I’ll have a go at creating a very basic sample application that you can pickup and play around with. I’ll even extend the framework just a little bit to implement a usecase I have that is not covered by the project itself.

Introduction

Spring mobile has it’s own home, you can find the homepage here. The project has a sample project as well. So why did I write this blog? Just to understand the project better and see how it fits in my personal projects. Maybe others will get a better understanding as well by reading this blog. Let me know if you find something interesting or have some improvements.

The sourcecode is available at Github : http://github.com/jettro/GridshoreSamples.git

The sample

I wanted to keep the sample as simple as possible to show how spring-mobile works. The sample must be complete enough to be able to explain the features that are important. Therefore I do not really create a functional thing. Just one page that shows whether you are using a mobile browser and some information about the browser you are using.

Simulation mobile platforms

Within Safari it is easy to pretend you are using another client than the safari browser that is installed. Use the developer menu bar item. Look at the following screendump.

How does it work?

To understand how the resolvers can do what they do, resolve the client and learn about it capabilities, you need to know a bit about an html request. I take it that all the readers of my blog have the basic understanding of a html request header. Each request has a body and a header. The header contains some meta data about the client. To get the basic understanding, look at the code of the LiteDeviceResolver. This resolver uses a few items from the header like User-Agent, OperaMini and wap support to determine if a client is a mobile client. The class has a number of keywords that it checks for presence in the User-Agent string. In our case using the iPhone browser, there would be a match for the keyword phone.

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7

Building the application

I use gradle to build the sample. Using the intellij and jetty plugin, gradle works like a charm. I do not want to go into to much details about gradle. You can find the file in the provided git repository.

Most important thing for the build configuration are the dependencies for spring mobile and an important library I use, wurfl. More on that later.

compile "org.springframework.mobile:spring-mobile-device:1.0.0.M1" runtime "net.sourceforge.wurfl:wurfl:1.2"

Spring mobile configuration

Spring mobile is about resolving the client or device used to send a request. Therefore configuring spring-mobile is about configuring the device resolver. There are two options. The default makes use of the LiteDeviceResolver. The second option makes use of the WurflDeviceResolver. More on these resolvers follows. For both configurations you just need one bean. The following code blocks show both cases.

That is most of it already. Now you can resolve the different devices. There is one more thing if you want to actually do something with a the obtained device information. You might want to inject the Device into the Controller. From the reference documentation I copied the following class that registers WebArgumentResolver with the controller invocation. I do not really understand why this is not available as a provided bean. I think this will come, or maybe I misunderstood something. For completeness I’ll show you the code as well.

CustomWebArgumentResolverInstaller

@Component
public class CustomWebArgumentResolverInstaller {
    @Autowired
    public CustomWebArgumentResolverInstaller(AnnotationMethodHandlerAdapter controllerInvoker) {
        WebArgumentResolver[] resolvers = new WebArgumentResolver[1];
        resolvers[0] = new DeviceWebArgumentResolver();
        controllerInvoker.setCustomArgumentResolvers(resolvers);
    }
}

Device resolvers

Device resolvers use headers from the request to determine the Browser. The LiteDeviceResolver and with it the LiteDevice are created using the device resolving technique as copied from wordpress. It does not expose any capabilities of a mobile device, just that you are dealing with a mobile device.

The Device interface only exposes the isMobile method. You can also use the toString method. The following jsp shows some very basic lines of text.

Is Mobile :

Device:

Spring mobile comes out of the box with support for wurfl, to be honest, I had never heard of it before. But it is a pretty extensive library that knows about a lot of phones and other mobile devices. It knows a lot more about the capabilities of your devise than the default LiteDevice. Spring mobile also provides namespace support for other device resolving strategies. The wurfl device resolver has the option to supply the data in a zip file and a patch in an xml file. The following image shows you part of the toString output of the same jsp. I did not use the patch functionality.

The WurflDevice implementation exposes a lot more properties. You can ask for all capabilities and the markup. The following jsp code is used to show additional information.

<%@ taglib prefix="c" uri="http://java.sun.com/jstl/core_rt" %>

Is Mobile :

Basic Device info

id
UserAgent
MarkUp

Capabilities

key	value

The screen than looks like this

iPad and other mobile devices with big screens

Some time a go I wanted to browse to the website of Schradinova using my iPad. I was redirected to the mobile website, but the mobile website was not available yet. There was no way to go to the actual site. By now the mobile site works, but still I cannot go to the original website using the iPad. Therefore I want to present the option to the user to explicitly go to the mobile site or to the normal site.

One way to accomplish this is by providing a mechanism to read a request parameter before doing the client resolving. Therefore I go back to my most simple example of showing the toString of the device. I have created a subclass of the spring provided DeviceResolvingHandlerInterceptor. My interceptor looks for the request param mobile. If found a dummy device is created with the isMobile method implemented. Might not be the nicest solution, but it does show the possibilities. The next code block shows the new HandlerInterceptor and the dummy device which is a direct copy of the LiteDevice.

public class WithOverruleDeviceResolvingHandlerInterceptor extends DeviceResolvingHandlerInterceptor {
    public static final String mobile = "mobile";

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        if (null != request.getParameter(mobile)) {
            boolean mobileOverruled = Boolean.parseBoolean(request.getParameter(mobile));
            Device overruledDevice;
            if (mobileOverruled) {
                overruledDevice = DummyDevice.MOBILE_INSTANCE;
            } else {
                overruledDevice = DummyDevice.NOT_MOBILE_INSTANCE;
            }
            request.setAttribute(CURRENT_DEVICE_ATTRIBUTE, overruledDevice);
            return true;
        } else {
            return super.preHandle(request, response, handler);
        }
    }
}

public class DummyDevice implements Device {

    public static final DummyDevice MOBILE_INSTANCE = new DummyDevice(true);

    public static final DummyDevice NOT_MOBILE_INSTANCE = new DummyDevice(false);

    public boolean isMobile() {
        return mobile;
    }

    public String toString() {
        StringBuilder builder = new StringBuilder();
        builder.append("[DummyDevice ");
        builder.append("mobile").append("=").append(isMobile());
        builder.append("]");
        return builder.toString();
    }

    private final boolean mobile;

    /**
     * Creates a GenericDevice.
     */
    private DummyDevice(boolean mobile) {
        this.mobile = mobile;
    }
}

I also add a link to the page to go to the mobile or the normal equivalent of the page. The following screendumps give you the different options using a mobile browser, a normal browser and using the explicit request for a mobile site or the normal site.

We start with the normal browser doing a normal request. Than we push the mobile link and after that the normal link. Look at the url bar, the is mobile and the first part of the description. Without an explicit preferred site it will show LiteDevice, in the other case it shows DummyDevice. Now three screendumps.

If you really want to see it work for a mobile browser I’d urge you to checkout the sample and try it yourself

Last remarks

Well, that is it. You should be capable now of creating your own mobile site. Finally I want to mention that spring mobile also has an option to redirect a user with a mobile browser to a completely different site. Check the reference manual to see how that works.

If you got this far, thanks for reading. See you next time.

Creating a w3c validated rss feed using Rome and spring 3

jettro — Tue, 16 Feb 2010 11:16:00 +0000

For my current customer I had to create an rss feed. In the java domain you immediately grab Rome to do the job. There was a catch. My customer wants (with good reason) to have feeds validated by the w3c feed validator. This turned out to be a slightly more complicated job. Luckily Rome has good support for extensions, so at least it was possible.

In this blog post I describe the challenges I had creating the validated feed. If you want more in depth information please check my post on my employers blog : serving a heavy load rss feed with spring 3 and ehcache.

Rome extension

At the moment I am missing the atom namespace in the resulting xml.

xmlns:atom="http://www.w3.org/2005/Atom">

I also miss the actual atom link:

Now we must tell Rome how to generate this content. Rome supports a plugin structure. Detailed information can be found on this page:

http://wiki.java.net/bin/view/Javawsxml/Rome05TutorialSampleModule

An extension consists of a module containing the data to be used by the extension, a generator if the extension needs to add items to the generated feed and a parser if the extension is about reading feeds. In my case I only need the module component and the generator. Finally you need to tell Rome about the extension, this is done in a configuration file that needs to be available on the classpath.

Module

The only data I need is the link for the atom element, the href. The module itself consists of an interface and an implementation. The interface defines the getters and setters. The implementation has a few additional methods. The following code block shows them both.

public interface AtomNSModule extends Module {
    public static final String URI = "http://www.w3.org/2005/Atom";
    String getLink();
    void setLink(String href);
}

public class AtomNSModuleImpl extends ModuleImpl implements AtomNSModule {
    private String link;

    public AtomNSModuleImpl() {
        super(AtomNSModule.class, URI);
    }

    public String getLink() {
        return link;
    }

    public void setLink(String link) {
        this.link = link;
    }

    public Class getInterface() {
        return AtomNSModule.class;
    }

    public void copyFrom(Object obj) {
        AtomNSModule module = (AtomNSModule) obj;
        module.setLink(this.link);
    }
}

The code is not to hard to understand. Concentrate on the copyForm method. Here we put the data from our model into the object that is used by the framework to extract data. This is why you need the interface as well.

Generator

Here we add the elements to the generated xml. JDom is used to generate xml from an object tree of Elements. The following code block shows the complete generator.

public class AtomNSModuleGenerator implements ModuleGenerator {
    private static final Namespace ATOM_NS = Namespace.getNamespace("atom", AtomNSModule.URI);

    private static final Set NAMESPACES;

    static {
        Set nss = new HashSet();
        nss.add(ATOM_NS);
        NAMESPACES = Collections.unmodifiableSet(nss);
    }

    public String getNamespaceUri() {
        return AtomNSModule.URI;
    }

    public Set getNamespaces() {
        return NAMESPACES;
    }

    public void generate(Module module, Element element) {
        AtomNSModule atomNSModule = (AtomNSModule) module;
        Element root = element;
        while (root.getParent() != null && root.getParent() instanceof Element) {
            root = (Element) element.getParent();
        }
        root.addNamespaceDeclaration(ATOM_NS);

        Element atomLink = new Element("link", ATOM_NS);
        atomLink.setAttribute("href", atomNSModule.getLink());
        atomLink.setAttribute("rel", "self");
        atomLink.setAttribute("type", "application/rss+xml");

        element.addContent(0, atomLink);
    }
}

Focus on the generate method. In this method we add the namespace declaration to the root element and we create a new element that is added to the provided element. The provided element is the channel element. The result of the feed now looks like this.

...

Configuration

The last part is telling Rome to use the new component. This is done by putting a rome.properties file on the classpath with the following contents.

rss_2.0.feed.ModuleGenerator.classes=nl.gridshore.rss.romeextension.AtomNSModuleGenerator

It is important to specify the rss_2.0 because multiple generators are available and this is the one that is used.

Encoding – response type

Another problem I had had to do with encoding. You can set the type of encoding in the generated xml, but the response of the spring controller/view combination does not follow this. Therefore you have to set the encoding of the response object explicitly. In the view component I added the following lines of code:

    @Override
    protected void prepareResponse(HttpServletRequest request, HttpServletResponse response) {
        super.prepareResponse(request, response);
        response.setCharacterEncoding(CharacterEncodingConstants.UTF8);
    }

I hope this can help others to create better feeds as well, suggestions for improvements are of course welcome.

When good guys start looking like bullies….

Ben — Tue, 23 Sep 2008 21:42:40 +0000

An opinion on SpringSource’s new maintenance policy

Introduction

On September 17, 2008 SpringSource announced a new Maintenance Policy regarding its software offerings, particularly its primary product (the Spring Framework) both in its open source and Enterprise editions. The new policy reads as follows:

Customers who are using SpringSource Enterprise, available under a subscription, will receive maintenance releases for three years from the general availability of a major new version. These customers receive ongoing, rapid patches as well as regular maintenance releases to address bugs, security vulnerabilities and usability issues, making SpringSource Enterprise the best option for production systems.

After a new major version of Spring is released, community maintenance updates will be issued for three months to address initial stability issues. Subsequent maintenance releases will be available to SpringSource Enterprise customers. Bug fixes will be folded into the open source development trunk and will be made available in the next major community release of the software.

What does the new policy mean?

So what does this new policy mean for developers and projects that rely on the Spring Framework? Well, it depends a lot on who you are and what your circumstances are.

For starters, it is important to note that this new policy does not mean that Spring Framework has suddenly ceased being an open source project. It is still open source software, published under the Apache License (version 2.0) and available for download by anybody (from the CVS repository). It will also remain possible to submit patches, build your own release, et cetera.

The changes brought about by the new policy, in practice, will only affect how prepackaged spring jars will be released by SpringSource. SpringSource will continue to release prepackaged jars of new major and minor versions (spring-project-x.0.0 and spring-project-x.y.0). In addition, SpringSource will provide prepackaged jars with bugfixes on those released versions for three months after the version is released. After that the new bugfix jars will only be available to people with an Enterprise license. Other people will have to check out bugfixes from the trunk and rebuild themselves, patch their existing libraries with patches from the trunk as needed or wait for the next major/minor release.

For most people the new situation will not make that much difference — most open source developers can simply rebuild or wait. The biggest pinch will probably be felt by commercial projects with restrictive corporate policies on upgrading libraries. Those projects will have to push their business customers to purchase Enterprise licenses if they need access to frequent bugfixes releases.

Is this fair?

The next major question is: is this new policy in any way fair? Is it not a gross violation of open source principles? Should open source software not be free?

Well, the first thing to remember here is the old adage about free software as formulated in the GNU’s Free Software Definition: the “free” in free software means free as in free speech, not free as in free beer. There is nothing in the open source philosophy, nor in practically any open source license that forbids charging money for software — as long as you make the source code freely available. Open source or free software is not a contradiction of any business model except the “trade secret” model. Making money off of software, related services (like prepackaging) or licensing models is absolutely allowed. If it wasn’t, open source would not have caught on. Can you imagine a Linux distribution like OpenSuSE evolving in a situation where you cannot charge for the software at all?

Another point that SpringSource raises (and it is a very valid one) is that they put a lot of effort into those bugfixes. The first three months of fixes always address stability issues (and those will remain available to the general public) but after that the problems noticed pretty much always move into the area of non-critical but still annoying issues. That’s not exclusive to Spring by the way, pretty much every project has a similar pattern with a different timeframe. Most projects (open source and commercial) address these issues by incorporating fixes into the new development version, with a patch of some sort for issues that are particularly bad. SpringSource has always done that as well, but has also “backported” bugfixes into intermediate bugfix releases while having a relatively wide spacing between major releases. That policy is taking more and more effort as more people use the framework (so SpringSource feels) and is also taking up a lot of resources that SpringSource wants to commit to other parts of their portfolio. So they have decided to stop the “backporting” for free and become a more “normal” OSS project with respect to bugfixes for the OSS community.

That aside, all open source licensing (like any other licensing) is a matter of intellectual property rights (specifically copyright). So even if the license on the Spring Framework had made it mandatorily free, SpringSource could at any time have chosen a different license and we’d be in the same position.

So, all in all, no, it is not unfair.

So why does it feel wrong?

And that’s just the problem here, isn’t it? Even though SpringSource is technically in the right at every turn, the new policy just feels wrong. Why? Probably because it feels like SpringSource is taking something away from the community. It feels like we had something for free and now we have to pay for it (even if we didn’t make frequent use of the free thingy when it was free and even though we can still get the free thingy with a little more effort). In other words, SpringSource has created itself a little public relations problem.

So is that bad? Possibly. Not in the sense that it will kill the Framework (Spring has a lot of traction and I have yet to see a framework that combines the three staples of Spring as well). But it will make people start looking at SpringSource in a different light and possibly look a little harder at alternatives to the SpringSource offerings. After all, SpringSource is no longer the Unlimited Fount of All That Is Good — SpringSource now has a nasty, commercial side that wants to make money and is not afraid to shortchange the community to do it (or at least, so it will now seem to quite a few people).

Where I think SpringSource went wrong with their new policy is in the area of presenting themselves as a trustworthy group of people. Let me explain: the open source community has a large problem with confidence when it comes to acceptance by traditionally commercial software developers. Commercial softwarehouses worked for decades in the classical model of secrecy, escrow, copyright and “what belongs to me is mine and keep your filthy paws off”. The idea that you gain more through openness and contribution, despite centuries worth of experience in science, is relatively new to software development outside the universities. The Internet, primary vehicle of OSS development, only started becoming popular in 1995, thirteen years ago. And OSS as a term is even younger. Most commercial entities are still getting to grips with the notion. A large part of such a change is trust and trust like that takes time to build. And when big players like SpringSource start doing things like this new licensing policy, it doesn’t help the trust along at all.

Let me give you another example of where a well-intentioned project went off the track due to mishandling the way people viewed it: the GNU General Public License. The GPL was a mainstay of the open source community for years in version 1.0 and 2.0. Pretty much every OSS and FOSS project adopted the GPL because it embodied the essence of OSS. Then came GPLv3 (around the time that OSS started getting general attention) and that… Well, “flopped” is probably too strong. But it was not an enormous success. So why is this? Is it because of digital rights management or because GPLv3 makes it mandatory to give away all your software for free and kill Bill Gates? I personally don’t think so. I personally think that GPLv3′s biggest problem is one Richard M. Stallman, who has a nasty habit of going around and claiming that everything that has ever remotely been near any open source software must now be open source (even if it plainly isn’t) and screaming everything that commercial software developers really don’t want to hear. Public presentation and public relations, in other words.

What to make of this… except a bonfire?

So, to recap: SpringSource is technically in the right, but I think they may have lightly shot themselves in the foot with this one. What kind of damage are they looking at? For starters, I think they may have made the battle for corporate acceptance a little more uphill than it was. Worse, I think they’ve lost themselves a lot of community respect — not by actually being bad, but by being ham-handed in their approach. At worst they may have opened the door for some community members to start looking elsewhere.

Ironically, of course, that presents an opportunity of sorts for SpringSource as well. Over the past year and a half Spring has become sort of a de facto standard for enterprise Java development. And being a standard can be dangerous, since it can lull you to sleep. If SpringSource has now accidentally created a chance for its rivals, that means SpringSource may soon have to compete on pure quality again. And what better way to earn (back) your OSS credentials than that?