This post I will talk you through a grails application I have created for a project of mine. I also take you through some of the things I learned last week. Finally I’ll show you that you can create a google app engine application using grails. There are ways to create grails applications, I’ll show you how to do it using the latest intellij 9 beta release.

Training

As mentioned in the introduction I attended the spring source groovy and grails training. The training was given by Peter Ledbrook, author of grails in action. I am not going to give a detailed description of the training. I did like the training, it gave a good overview of the possibilities of grails as well as the groovy language. And yes, I am now very interested in the options of groovy. Especially for creating web based applications with grails, groovy just shines. Not sure when I should use groovy and when normal java. Have to explore to find out. Some of the things I liked about groovy are:

• Assignment of parameters that are evaluated immediately ${age} or when used ${->age}
• Using negative indexes in arrays: [1 .. -2] strips of the first and last item from an array
• Groovy truth, explicit use of values in a comparison. null, 0, “” and [] are all false in a comparison
• Closures, still hard to determine when to use a closure and when a method. But closures are used very often in groovy.
• Out of the box test integration with things like: MockFor, Expando, and as a plugin GMock.

Of course we learned a lot more than this, check the training or the book yourself if you want more. Next up was grails. I did look at grails before at grails, but tool support was pretty limited back than, and the plugins were not really widely available. A lot has changed since than. SpringSource came out with grails support in SpringSource Tool Suite, and Intellij has grails support as well. To be honest, intellij was much more productive than STS. Actually STS was almost not useable. Since it was the first release with grails support it will most probably improve the coming releases. Intellij worked very nice out of the box (I did use the beta version of intellij 9. Some of the features I liked during the training are:

• Gorm, I must mention this Groovy/Grails Object Relational Mapping framework first. This is very important for grails and why it is so productive. I just love these dynamic finders (more later).
• Plugin architecture, the easy of installtion and usage of plugins is very nice. I like the db-util plugin for its simplicity and the help when working with internal memory databases. A killer plugin during development.
• Scaffolding, for auto responding to CRUD like operations.
• flash parameter to be used for adding something to the session for just one request. Ideal for passing messages.
• Using templates, ideal for doing ajax calls and reuse of small view elements.

Oke time to get our hands dirty, next up is a short introduction in the sample I am going to create. Than some code (which is acutally not that much)

The application to create

For a customer at JTeam, we have a large amount of servers that we need to prepare. The actual preparation is mostly done by the hosting provider. Sometime we have no idea what the status is of some servers. We started to use an excel sheet to communicate the status among our team. This is not ideal and I decided to create a small grails application for this. It’s all very basic, but useable.

Every server has a status, some comments and an external ip address. Each server is also part of a number of vlans and belongs to an environment (dev,test,accep,prod).

The code

The code is available online at github. Yes I know, it is not at google code. I got extra motivated to start using git since the training. Peter was working so flexible with his source code, I want that too. The best way to learn something is by doing it. Therefore I create a git repository at github. You can find out more at the following url. There is a very easy way to download the complete source code without using git, check the page.

http://github.com/jettro/MyServerPark

The basics

Starting a grails project is very well documented on the grails website. Except for one controller, the OverviewController, all controllers use: def scaffold = true. Which means I don’t write any code. Also not view components. I did create some domain classes that have relationships with each other. The next sections cover the different elements of the application.

The domain model

Some things to know about the domain model:

• belongsTo is used to declare a relationship of many-to-one, the next line gives an example for Comment. This gives the comment a property server.

static belongsTo = [server:Server]

• The comment class also gives meta data about the mapping and the type used for the database as well as constraints. In the example we create a field for larger amounts of text.

class Comment {
    String name
    String content

    static belongsTo = [server:Server]

    static mapping = {
        content type: "text"
    }

    static constraints = {
        name(blank:false, maxSize:50)
        content(blank:false, widget:"textarea")
    }
}

• The previous code block also shows one of the things that might be a bit strange. With widget:”textarea” you indicate the type of field to be used at the front end. Might be a bit strange in a domain model.
• A one-to-many relationship is also easy to implement, check the following line from Environment:

static hasMany = [servers:Server]

Check the code for more examples of relationships.

Scaffolding controllers and views

Grails makes use of scaffolding, this can be static and dynamic. In the static case, code is being generated and can be altered. The dynamic case is evaluated runtime. In this project I tried to use dynamic scaffolding as much as possible. Usability might not always be what you want. In that case you should use static scaffolding or create something yourself. I managed to do well with the scaffolding on dynamic (except for the google app engine sample). But let us focus on the customer controller.

The overview controller

Grails makes use of convention over configuration. This is also true for controllers and views. The overview controller listens to the /overview url and points to the overview.gsp file. The following code block gives the complete OverviewController groovy class.

class OverviewController {
    def show = {
        redirect(uri:"/server/show/${params.id}")
    }

    def comments = {
        def currentServer = Server.findById(params.id.toLong())
        def allComments = Comment.withCriteria {
            server {
                eq("id",currentServer.id)
            }
        }
        render(template:"/shared/comment",model:[comments:allComments, server:currentServer])
    }

    def index = {
        def environments = Environment.list()
        [environments:environments]
    }
}

There are three methods in the controller. The show does a redirect to the show method of the server controller. The index returns list of all environments and uses the view index.gsp. The comments method is a different cup of thee. This responds to an ajax call and returns a fragment. focus on the last line for now. You can see we use the render call. This makes use of a template. Using the render call, we obtain the html and push it back to the client. The section about views will give a bit more detail. Next up are the dynamic finders

Dynamic finders

The previous code block gives a few examples of finders. In index we obtain all the environments with Environment.list(). This is the most basic finder. Another finder is in the comments method. There we obtain the server by an id. The call is findServerById and you provided an id. Another calls could be: findByFunction, findByFunctionLike “%database%”, findByFunctionIlike “%DataBase%”. You can also make combinations with And/Or and multiple parameters. Pretty cool.

The comments method also shows the Criteria API for looking up all comments for a specific server. These type of queries, limited by a related item, are not supported by the dynamic finders.

Gsp for views

Creating the views is not very hard. If you want to learn how it works, I suggest generating the scaffolding views. I want to focus on something that is just a little bit more advanced. Ajax calls. In my project I want to obtain all comments for a server using an ajax call. In the overview page we use a special tag from grails, the remoteLink. This tag does an ajax call to the server and prints the result in the div with provided id.

<g:remoteLink action=”comments” params=”[id:server.id]” update=”comments”>comments</g:remoteLink>

As you can see, we provide the server.id and we update the div with id comments.

This does not work out of the box, you must specify the prototype javascript library to be included. You can do this in the file:

views > layouts > main.gsp

Add the following line:

<g:javascript library=”prototype” />

Now everything should be fine and you can test the application. A command like grails run-app should be enough.

Documentation

If you want more information about grails, try the following references:

• http://grails.org/doc/1.1.x/
• http://groovy.codehaus.org/groovy-jdk/
• http://www.manning.com/gsmith/
• http://www.jetbrains.com/idea/nextversion/index.html
• http://www.springsource.com/training/grv001
• http://www.springsource.com/products/sts

Grails and google app engine

Grails uses plugins for all additional functionality. There is also a plugins available for google app engine. There is already good documentation available for this plugin. Therefore I keep this short. The following code block shows the commands. For more information, check the website of the plugin:

http://grails.org/plugin/app-engine

# grails create-app share-this
# grails uninstall-plugin hibernate
# grails install-plugin app-engine
choose jpa when asked
# export APPENGINE_HOME=<path to your sdk install of google app engine>
# grails app-engine run
# grails set-version 1
# grails app-engine package
# $APPENGINE_HOME/bin/appcfg.sh update ./target/war
# grails install-plugin gorm-jpa

Than I used intellij to create some domain classes and controllers. Take the following into consideration.

• Put domain classes in a package or it won’t work
• Put Controller classes in the same package
• generate views, the auto scaffolding seems not to work.

You can look at the result here : http://share-this.appspot.com/. It could well be that the app does not start the first time, a refresh might help. I’ll have a look at this in the future. There is a problem with the CPU consumption and my grails application.

Read on to find out what I did.

Google app engine uses the concept of static files. This is a performance optimization. Using the file appengine-web.xml you can configure the way google handles static files. You can include and exclude certain files using their extension or name. More information can be found here at google. This all works nice in the online version, however there seems to be a problem with the development server. Some solutions try to configure the local version as well, still that did not work for me. I decided to look for a servlet that serves static files. Yes it can be that easy.

Check the following post serving static content using a servlet. This post gives you a servlet implementation to serve static files. I used the StaticServlet.java, read the post for more information. Adding this servlet to your web.xml solves the problems in your local environment.

    <servlet>
        <servlet-name>default</servlet-name>
        <servlet-class>org.yourscrum.web.StaticServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>default</servlet-name>
        <url-pattern>*.png,*.jpg,*.css,*.js</url-pattern>
    </servlet-mapping>

That is it, now you can test your stuff locally and all your scripts, images, styles are loaded by your application. Of course you have to remove this servlet before uploading your application.

Hope it helps people with their local debugging of jquery scripts or other javascript things. Speaking about jquery, the next post will be about a component used for sorting items using the jquery ui component. I had an issue adding an image to the items in the sortable component.

Quick intro

For an eclipse user there are other steps to get things done. There is something like a plugin that might work depending on other plugins that you have installed and some eclipse magic. Sorry, I am a non believer. I prefer another IDE.

So for the non eclipse guys you need to download a zip file with all the resources you need to start working with the new sdk. I create a symbolic link to the downloaded version, that way it is easy to change for future releases.

ln -s appengine-java-sdk-1.2.0/ appengine-java-sdk

As the getting started suggest, you can now test your infrastructure with one of the provided demo’s.

./appengine-java-sdk/bin/dev_appserver.sh appengine-java-sdk/demos/guestbook/war/

Open your browser and go to the link http://localhost:8080/. You should see the following screen.

Oke, time to take the next step. The structure for an application is the same as for a web archive. There is a catch, you cannot use the war file itself, you need the exploded directory. Luckily it is very easy to do this with maven or with Intellij. Those eclipse users that are still reading this post have an easier life. They can use the wizard from the google plugin. I start off with a normal intellij java project with a web facet.

Google app engine works with servlets, so we need to create a servlet, we also need to add this servlet to the web.xml. Since the code I used is at the google tutorial, I’ll not repeat it here.

Google appengine needs an additional xml configuration file. This file is called and it looks like this:

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <application></application>
    <version>1</version>
</appengine-web-app>

Next step in intelliJ is to make the module, you can find the exploded war in the folder out > exploded > GoogleAppEngineGuestbookWeb. Using this exploded folder you can start the appengine. We need a runner in intellij. This is not very hard to do.There are two important steps. The first step is to create a library that you add to your module. This library contains one jar file : appengine-tools-api.jar. You can find this jar in the distribution of google appengine you downloaded. If you do not know how to add a library, open the module settings (apple: command + . Create a new library and when asked say yes to add it to your module. The screen should look like the following image.

Next up is configuring a runner. Create a runner of type Application. Now use the script dev_appserver.sh, that you can find in the distribution of google appengine in the bin folder. We need to set the environment variables, the program parameters and the main class.

Main class

com.google.appengine.tools.KickStart

Program parameters


com.google.appengine.tools.development.DevAppServerMain
out/exploded/GoogleAppEngineGuestbookWeb


Check the second parameter, this is the location of the exploded war file that intellij creates for you.

Environment variables


SDK_LIB=YOUR_GOOGLE_APPENGINE_HOME/appengine-java-sdk/lib
SDK_CONFIG=YOUR_GOOGLE_APPENGINE_HOME/appengine-java-sdk/config/sdk


The screen for configuring your runner now looks like this

That is about it, now you can run the application. After running, the output in the console is:


2009-04-08 23:53:44.332 java[3557:10b] [Java CocoaComponent compatibility mode]: Enabled
2009-04-08 23:53:44.333 java[3557:10b] [Java CocoaComponent compatibility mode]: Setting timeout for SWT to 0.100000
The server is running at http://localhost:8080/


Now you can point your browser to the following url : http://localhost:8080/guestbook and Hello world should be yours.

Deployment

Before you can upload, you need to register yourself with google. If you already have an appengine account, you can request the java beta admittance. After that create a new google appengine application in your domain. That’s it, remember the name of the app you just registered because you need it in the next step.

Next up is the actual deployment, this is not a very complicated process. You can use the same directory that you use for the test server. Be sure to change the file appengine-web.xml. Add your name of the application to the application tag. If your using intellij do not forget to make the module. In my case this application name is doingjava. Now all you need during the upload is your google account name and password. Next is the command to do the actual upload


./appengine-java-sdk/bin/appcfg.sh update PATH_TO_YOUR_INTELLIJAPP/out/exploded/GoogleAppEngineGuestbookWeb


In my case you can check if it works on the following url: http://doingjava.appspot.com/guestbook.

Conclusion

This is not a conclusion on the usage of google appengine and java. For me it was an exercise to create a project in intellij and be able to use the google mock webapp runner. This seems to work nicely. The future will bring more posts about google appengine which now support java.

I do want to stress this is not a best practice, I advise to use an ant script that you can get from google at the following url. http://code.google.com/appengine/docs/java/tools/ant.html

Some important url’s

• http://www.gridshore.nl/2008/10/09/using-google-appengine/
• http://appengine.google.com/
• http://code.google.com/appengine/docs/java/gettingstarted/
• http://code.google.com/appengine/docs/java/tools/ant.html

Read on if your are interested in the possibilities of google apps.

I do want to give a warning, this is my first google apps experience and on top of that also my first Python experience. Do not take this code as a best practice, just learn about the possibilities.

Starting your first google apps engine application starts by downloading the right software and registering. By registering yourself, you get the rights to register and deploy a certain amount of applications. At the time of writing it is ten applications. If you do not want to pay, there are some limitations in the amount of storage that you get and the amount of visits your application can receive. It is a fair way of setting your first steps, I think it will take a while before you reach your limits. Unless you want to store images, documents or videos within your application. If that is what you are up to, you are better of using the other services of google and than integrate with these services. There are apis for google docs, images, mail, calendar and lots of other services. That was one of the reasons I wanted to try this integration.

During the JAOO I followed the following tutorial, it took me around an hour including downloads and uploads. This is a nice way of getting up to speed.

http://code.google.com/appengine/docs/gettingstarted/

Time to introduce my application, I am going to create an application that shows all your documents within google docs. Sounds impossible? Well it isn’t. Let’s start with the basic elements of the application:

• app.yaml – config file of the application redirecting all calls to the mydocs.py script.
• main.css – just like you think, a stylesheets in the static stylesheets folder
• index.html – template file with html and placeholders
• mydocs.py – the python script containing all important code

For now, focus on the python script. The first time I show this code, it contains only the generation of the login or the logout link.

import os
from google.appengine.ext.webapp import template
import cgi

from google.appengine.api import users
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

class MainPage(webapp.RequestHandler):
  def get(self):

    if users.get_current_user():
      url = users.create_logout_url(self.request.uri)
      url_linktext = 'Logout'
    else:
      url = users.create_login_url(self.request.uri)
      url_linktext = 'Login'

    template_values = {
      'url': url,
      'url_linktext': url_linktext,
      }

    path = os.path.join(os.path.dirname(__file__), 'index.html')
    self.response.out.write(template.render(path, template_values))

application = webapp.WSGIApplication(
                                     [('/', MainPage)],
                                     debug=True)

def main():
  run_wsgi_app(application)

if __name__ == "__main__":
  main()

What can you see in this code? First of all, some imports. We use the users object and the special webapp and a templating engine. Then you see a class called MainPage. This class responds to the root, as is configured in the construction of the application using the special webapp method WSGIApplication. The final part is the actual runner. The MainPage class contains code to check if the current is logged in and it changes the text and the presented link based on this information. You can also see the templating in action. The values are set, then the template file with placeholders is chosen and finally the output is rendered.

If you want to check out the sources of my application, go to my google code project.

http://code.google.com/p/gridshore/

With all the files in place, we can start our first test run. Google provides a test environment, there are some facilities for mocking a login form, the database, and other things as well. Another thing that comes out of the box is a special admin console. Here you can get more information about what happens in your application and you can issue python command in a command line environment.

Run the application using the following command and go to the two provided urls. The image presented is the mentioned admin console.

dev_appserver.py jettrodocs – Starts the developement application server.
http://localhost:8080/ – locally running application
http://localhost:8080/_ah/admin/ – admin console

Now we upload the application for the first time. In the command jettrodocs is the directory that contains all my sources. Therefore you must issue the command one directory higher than your sources.

appcfg.py update jettrosdocs

My application is now deployed to the live google environment. Next I’ll give you the url. As a developer of your own applications, you also have access to an online dashboard with all different kind of information about your running application. This is also the place where you can register your application.

http://docsjettro.appspot.com/ – The running application
http://appengine.google.com/ – The dashboard for all your apps

Up till now, I have shown you the hard way of starting the application, uploading your application, typing the url’s to go to your local console and the online dashboard. Basically because I use the terminal for a large part of my day, I feel comfortable with it. There however a nice other way. That is the GoogleAppEngineLauncher. This small app enables you to start, stop, upload your application by pushing some buttons. One nice thing is that you can run your apps on different ports easily, so you can keep on developing them at the same time. The following images gives a screen dump of this google app engine launcher.

The final step is getting the document retrieval working. This turned out to be more difficult than I expected. There is a lot of information to be found on the web, some very good documents as well as nice examples. In the end I managed to get it working. I’ll focus on the bits that were harder for me to grasp. be sure to check out documentation from google. Check the references at the end for a few of the articles I used to get this going.

To be able to connect to google docs api, or feeds, your application needs to be approved by the user that runs it. The whole approval works with a token that is generated and than passed from the authentication service of google to your application. A user gets to see a special google screen that asks him to approve the access to the documents. Next to this approval, the user needs to be logged in as well. So without the user logging in, the application cannot see the documents. By default the mentioned token is only valid for 1 request. There is a mechanism to store the token in the session, that way it will be valid during the complete session.

When the user has authorized the application access to his documents and the user is logged in, the application can do a query for the documents and than print it on the screen.

Time to have a look at the code, for a complete view of the file, please click here. First let’s have a look at the get function that is called when a request comes in. We have seen most of it already, therefore we begin with the new stuff. The following piece of code analyzes the requested url and reads the token as well as the token_scope. To be honest, I think I could have hard coded this token_scope since I am only reading one feed. In the original example you could enter your own feed. Other than that the code speaks for itself.

for param in self.request.query.split('&'):
  # Get the token scope variable we specified when generating the URL
  if param.startswith('token_scope'):
    self.token_scope = urllib.unquote_plus(param.split('=')[1])
  # Google Data will return a token, get that
  elif param.startswith('token'):
    self.token = param.split('=')[1]

Next steps are authenticating the user and checking the session for an existing token. I’ll get back to that later on. Next step is do the actual call to the feed and read the documents from the feed. The feed is called in a separate method. When this method throws an error, the following code reads the error message and stores it in the template values collection. This error handling is done in the following piece of code.

documents=[]
responseMessage = ''
try:
  documents = self.FetchFeed()
except Exception, strerror:
  responseMessage = strerror

Finally the template is filled with values and outputted to the browser. That is pretty straightforward. Next step, the authentication.

def ManageAuth(self):
  self.client = gdata.docs.service.DocsService()
  gdata.alt.appengine.run_on_appengine(self.client)
  if self.token and self.current_user:
    self.UpgradeAndStoreToken()

def UpgradeAndStoreToken(self):
  self.client.SetAuthSubToken(self.token)
  self.client.UpgradeToSessionToken()
  if self.current_user:
    # Create a new token object for the data store which associates the
    # session token with the requested URL and the current user.
    new_token = StoredToken(user_email=self.current_user.email(),
        session_token=self.client.GetAuthSubToken(), target_url=self.token_scope)
    new_token.put()
def LookupToken(self):
  if self.feed_url and self.current_user:
    stored_tokens = StoredToken.gql('WHERE user_email = :1',self.current_user.email())
    for token in stored_tokens:
      if self.feed_url.startswith(token.target_url):
        self.client.SetAuthSubToken(token.session_token)
        return

Here you are introduced with the gdata library. This is a google project to enable developers to easier integrate the google apis using python. You add the required files to your project, check the references for a link to the download and the tutorial. Using this library it becomes easy to add the token to the session, you just call the method UpgradeToSessionToken. The token are stored in the database, that way you can create different tokens for different feeds. Again something I probably could have lived without. Querying for the right token is done in the method LookupToken. We query for the tokens for the current user and look for a token for the requested feed.

Now that we have the token, we can finally call the feed and handle the result.

def FetchFeed(self):
  # The following creation of the client is necessary for the case a client is not authenticated
  if not self.client:
    self.client = gdata.docs.service.DocsService()
    gdata.alt.appengine.run_on_appengine(self.client)

  try:
    return self.ListAllDocuments()
  except gdata.service.RequestError, request_error:
    # If fetching fails, then tell the user that they need to login to
    # authorize this app by logging in at the following URL.
    if request_error[0]['status'] == 401:
      # Get the URL of the current page so that our AuthSub request will
      # send the user back to here.
      next = self.request.uri
      auth_sub_url = self.client.GenerateAuthSubURL(next, self.feed_url,secure=False, session=True)
      raise Exception('<a href="%s">Click here to authorize this application to view the feed</a>' % (auth_sub_url))
    else:
      raise Exception( 'Something else went wrong, here is the error object: %s ' % (str(request_error[0])))

def ListAllDocuments(self):
  feed = self.client.GetDocumentListFeed()
  documents = []
  for i, entry in enumerate(feed.entry):
    doc = Document()
    doc.title = entry.title.text.encode('UTF-8')
    doc.author = entry.author[0].name.text.encode('UTF-8')
    documents.append(doc)
  return documents

The ListAllDocuments method does the actual call and uses the result to create new Document instances that are than provide to the template engine. The FetchFeed function does an interesting thing. It handles exceptions. If the user is not authorized an exception is thrown with a link how a user can authorize the application to read his documents as described before. In case of any other errors, the error is just returned to the screen.

The application is done, you can deploy it using the command line tool or the gui. The actual solution looks like this.

You can try it out yourself here

http://docsjettro.appspot.com/docs

References

• getting started – tutorial walking you through all the required steps to deploy a hello world application
• google docs protocol – manual for the protocol of interacting with documents
• google docs python – Manual for using python to interact with the google doc feeds.
• gdata python client – python client that acts as a wrapper around a lot of different google apis, including google docs and google authentication.
• using the google datastore – manual for using the google datastore from python.
• authentication module – documentation for the google authentication model