I started thinking about putting the sources in a version control system. Since I like Git and have a Github account I wanted to store the files in Git. But I had some other requirements.

• I do not want to manually copy the changed files to the server.
• I am limited to ftp access, the hosted does not provide ssh access.
• I want to be able to make quick changes from multiple laptops.
• I want to have the changes taken from Git but also changes that have not yet been committed if I wanted to.
• I have no requirements for removing files, this does not happen a lot.




Since I am on a Mac and have some very basic experience with Bash, I decided to use Bash. I did have some issues with FTP, but more on that later on.

If you are curious about the websites I have created:

Yvonne van der Mey ~ Wildlife Photographer

Nico Bulder ~ Wildlife Painter

Repository layout

I want to support multiple sites in one repository. I give each site it’s own folder. The root of the project contains the deploy.lib file. This file contains the reusable code, which is almost everything. Each website folder contains a deploy.sh file. In the website folder we also have a file ftpconfig. This file is discussed later on when discussing the NcFtp client.

I do not store the complete wordpress installation in git. I most of store the template that you can find under wp-content/themes/yvonne. But I also add files that I added to the wordpress install like the favicon.ico. The idea is that I copy all changes over the existing wordpress installation. Like I stated before, I am not going to remove files. Using this mechanism I can also add custom plugins in the same way.

Explaining Git commands

The sources are stored in Git. This makes it easier to track changes and it helps me doing stuff the right way. So what is the right way? I want to deploy files that have changes in commits. Therefore I store the sha1 key of the last commit that I have deployed in a file. The next time I deploy changes, I read this file and deploy all changes after this commit. Sometimes however, you are debugging things and you do not want to continuously do commits. In that scenario you want to be able to deploy all changes in your local copy. This is also possible with the script.

Changes based on commits

Git has good support for finding more information on commits. There are multiple commands that can do the trick. For the most clean output came from the following command:

git show --pretty="format:" --name-only $startingfrom..HEAD

The $startingfrom is the sha1 key obtained through a provided parameter or from the file that was written the last time we did a deploy.

SHA1 of the last commit

We want to keep track of the changes that have already been deployed to the server. Since we are basing our deploys git commits, we want to obtain the sha1 key of the last commit. Git provides are daily easy way to obtain this key.

git log -1 --pretty=format:%H > $shaversion

With these commands we know enough to obtain changes after the last deploy. But what if we are trying something and want to make it possible to deploy files that have not yet been committed.

Changes in the local workspace

Changes that are not yet committed are called changes in the workspace of git. Again provides a good way to obtain the names of the files than have changes in the local workspace. You can check them against he HEAD.

git diff HEAD --name-only

That is it. Now we have the power to create a list of files that have changed. How can we copy files to the server using ftp.

NcFtp

The bash script creates a folder with a nested folder structure. The structure contains files on multiple levels. We want to use one command to recursively ftp all files to the server. Well, that is a hassle. First you have to overcome issue with ftp and logging in. Using the .netrc file this is easy to overcome on the mac. Still this does not resolve all our issues. Ftp does not support recursive ftp in a folder. You can use mput, but this only works for files in a folder. Not in a nested folder.

There are other options for ftp. I decided to use ncftp client software. Easy to install using Homebrew or Macports. I store the ftp connection data in a file that I configure in the ncftpput command. THe following command recursively copies the sources in the provided local folder to the provided remote folder.

The ftpconfig file must contain three properties: host, user, pass. The following command shows how you can copy the content of a directory to the server.

ncftpput -f ftpconfig -R mainwebsite_html deploy/yvonne/*

Doing bash

Time to put the pieces together. The glue is provided by bash.

Site specific configuration

Most of the bash code is shared between the different websites. That code is in the deploy.lib file. Of course you can place all code in one file. I have just one parameter and the included of the library file in the specific site deploy.sh file.

sitename="yvonne"
. ../deploy.lib
exit 1

Handling input

You can provide a few options to the script to influence what it does. It is good practice to show the options in a usage or help method. The following code block shows this method as well as the handling of the provided arguments.

function usage() {
	echo "Usage: `basename $0` [--nocommit] {GIT REVISION SHA}"
	echo "    optional flags: "
	echo "        --help, --usage # show this file"
	echo "        --nocommit #By default only commits are used, with this flag we use changes in your workspace."
	echo "        --sha1 <value> #Uses the commits from the provided value to the HEAD. Default reads the sha1.txt file."
}

commitonly=true
while true; do
	# case $# in 0) usage; exit 2 ;; esac
	case $1 in
		--help)				usage; exit 1 ;; esac
	case $1 in
		--usage)			usage; exit 1 ;; esac
	case $1 in
		--nocommit)			shift; commitonly=false ;; esac
	case $1 in
		--sha1)				shift; startingfrom=$1; shift ;;
		*) break ;;
	esac
done

The default is to gather all changes since the last commit that got deployed using the sha1 key in the sha1.tx file. By providing the –nocommit flag you can tell the script to switch to gathering changes in the local workspace. By providing the argument –sha1, you can pass another sha1 key to obtain changes for.

Obtaining the right changes

To find the files that have changed, we create a file containing the path to a file that is changed per line. By looping through these lines we can copy all files that we want to deploy. The next code block creates this file, but it has two side effects. It also creates a version.php that contains the date of the deploy as well as all the files that have been deployed. The second site effect is creating the sha1.txt file that contains the sha1 key of the last commit that has been deployed.

filelist="files-$(date +"%s").txt"
mkdir -p "deploy/$sitename"
versionphp="deploy/$sitename/version.php"
shaversion="sha1.txt"

if $commitonly ; then
	if [[ ${#startingfrom} -lt 1 ]]; then
		echo "Need to reed the sha1.txt file"
		while read line; do
			startingfrom=$(echo $line);
		done < $shaversion
	fi
	echo "Using commits for changes: $startingfrom"

	git show --pretty="format:" --name-only $startingfrom..HEAD > $filelist
	echo "Committed changes: $(date)" > $versionphp
	echo "<br>Last commit: $startingfrom" >> $versionphp
	git log -1 --pretty=format:%H > $shaversion
	echo "" >> $shaversion
else
	echo "We only use changes in the workspace"
	git diff HEAD --name-only > $filelist
	echo "Local changes: $(date)" > $versionphp
fi

Line 6 determines if the starting from parameter has been provided by an argument to the script. We determine if the length of the value is smaller than 1. If so we obtain it from the file sha1.txt. I think this was the hardest trick.

Ftp the changed files

Now we have a file that contains all the names of the files that have changed. The next block loops over the files and filters a few of them based on the website we are working for at the moment. If we have files left at the end we use the ncftpput command to actually ftp them.

ftpfiles=false
for file in $(<$filelist); do
	if [[ $file == $sitename* && $file != $sitename/deploy* && $file != $sitename/sha1.txt ]]; then
		echo "$file"
		echo "<br>$file" >> $versionphp
		mkdir -p "deploy/$(dirname $file)"
		cp "../$file" "deploy/$file"
		ftpfiles=true
	fi
done

if $ftpfiles ; then
	ncftpput -f ftpconfig -R mainwebsite_html deploy/yvonne/*
else
	echo "Nothing to ftp."
fi

Concluding

I am now able to adjust my workflow. I can make the changed locally, commit them into git before I actually automatically deploy them to the server. It is nice to see what you can do with a little bit of bash. If you have improvements please let me know.

Incubator Status

The mongo event store is becoming fully functional right now. At the moment the event store is functional for the 1.3.x branch of axon. We have a working example “The Axon Trader” using the event store. We do not have it running in a project in production yet. If you are considering to do use it in production please let us know about your findings.

The planning is to drop the incubator status starting Axon 2.0. Using the mentioned sample we are going to do some stress testing and we are thinking about improvements. More on these in the Next steps section.

Connecting to Mongo

MongoFactory

Mongo can be used as a single instance, which is not really suitable for a production environment. A better setup is to use a replication set of mongo instances. Using the MongoFactory this can be hidden for the other parts of the application.

The factory takes a list of server addresses, you can override the MongoOptions as well as the WriteConcern. Of course we take good defaults. There is an option to set the mode to single instance or multiple instance using a system property: axon.mongo.singleinstance. This system property takes precedence over other settings.

In the end, the MongoFactory is used to create a Mongo instance. This instance should be provided to the implementation of the MongoCollections class. More on this later on.

MongoOptionsFactory

The configuration of Mongo using the java driver is executed using the MongoOptions class. This factory is used to create such an object using defaults but with the possibility to provider other values for the most important properties. Some of the options you can configure are: number of connections per host, the connection time-out and auto-connect retry.

The collections in Mongo

Mongo uses collections to store data. The names of these collections that Axon uses for the event store as well as the saga store can be configured. Of course logical default values are provided as well. If you want to use other collections you can provide your own implementation of this interface or provide other names to the default implementation of the interface.

Overview of the classes

The following image gives you an overview of the important classes.

Event repository

The event repository, MongoEventStore, is responsible for storing and retrieving the events. Just like the other implementations of the event stores, you can provide your own event serializer. By default XStream is used to serialize the events.

The java driver for mongo makes it easy to store new documents in Mongo and query the existing documents. By using the EventStoreCollections object we obtain a reference to the actual collection and by calling insert we add new documents to the collection. To get an idea about the code, have a look at the following function of the event store.

public void appendEvents(String type, DomainEventStream events) {
    List<DBObject> entries = new ArrayList<DBObject>();
    while (events.hasNext()) {
        DomainEvent event = events.next();
        EventEntry entry = new EventEntry(type, event, eventSerializer);
        entries.add(entry.asDBObject());
    }
    eventStoreCollections.domainEventCollection().insert(entries.toArray(new DBObject[entries.size()]));
}

A similar method is available to store snapshot events.

It becomes interesting when reading events to recreate an aggregate. Before reading events that need to be applied, we first need to find the latest snapshot. The following code block gives an idea about finding the latest snapshot event.

private EventEntry loadLastSnapshotEvent(String type, AggregateIdentifier identifier) {
    DBObject mongoEntry = BasicDBObjectBuilder.start()
            .add(EventEntry.AGGREGATE_IDENTIFIER_PROPERTY, identifier.asString())
            .add(EventEntry.AGGREGATE_TYPE_PROPERTY, type)
            .get();
    DBCursor dbCursor = eventStoreCollections.snapshotEventCollection()
            .find(mongoEntry)
            .sort(new BasicDBObject(EventEntry.SEQUENCE_NUMBER_PROPERTY, -1))
            .limit(1);

    if (!dbCursor.hasNext()) {
        return null;
    }
    DBObject first = dbCursor.next();

    return new EventEntry(first);
}

Now you have seen the main features of the event store in Mongo. The next step is to store and load Saga instances.

Saga repository

The mongo part for Saga’s is even easier than the events. We do not have to account for snapshots, which makes it easier. Just use the identifier of a saga instance to store, load and delete a document. The repository supports storing the associations between Saga’s as required by Axon. It also supports the ResourceInjector if your saga needs other resources to do it’s job. Think about the command bus to send new commands.

The structure is the same as for the event store implementation. The SagaStoreCollections provides access to the collections in the mongo database as well as the database itself.

Next steps

We are constantly looking for improvements in the mongo implementation. Allard is working on a specific Mongo serializer. That way we are nog serializing to XML and storing that xml. We will store the content in a DBObject and that way in a basic mongo document structure. More on this will follow later on in another blog post.

As mentioned we are planning on doing some performance tests and experiment with indexes to improve performance.

I you have improvements, bugs or ideas feel free to file an issue on our Github project.

http://blog.dutchworks.nl/2011/12/13/embedding-rss-in-hippo-using-the-pipelines-feature/

Enough of the fun part, now seriously. JTeam has outgrown it’s name. The past year a lot has changed and a new name is the next logical step. I cannot explain this better than my CEO Steven Schuurman. If you are interested you can read his announcement here:

http://blog.dutchworks.nl/2011/10/10/hot-off-the-press-as-of-today-“jteam”-will-be-known-as-“dutchworks”/

Head over to our new website: http://www.dutchworks.nl. Help us spread the word.

Hope to see you back when we write our next technical blog.

In most of our projects we use Artifactory. It is nice that artifactory comes with a REST based api. The url’s can be a bit better and the content type of the response is not what I would expect. Still the documentation is good and lost of actions are possible. In this blog post I show you the basics of interacting with the REST api using groovy. Now let us shrink the repo.

Introducing the artifactory api

The frog wiki has a wiki page dedicated to the api. This page shows the GET/DELETE/PUT requests that you can execute. It shows the url, the response and the content type. If you click the following link you jump to the definition of Folder Info. After the link an image with the same data at the time of writing.

http://wiki.jfrog.org/confluence/display/RTF/Artifactory%27s+REST+API#Artifactory%27sRESTAPI-FolderInfo

In the image you can also see the JSON response for the specific request. Have a look at the structure of the response. Look at the children. We will use it later on.

Obtaining folder info

Time to show some code. We will first have a look at a function to obtain information about the folder as provided by the caller. Since we group the functions in a class we have to look at configuration of this class as well. The constructor accepts a map that we call config. That way you can provided configuration for the server, the repository and the versions to remove. An example of the config map is.

def config = [
    server: 'http://localhost:8080',
    repository: 'libs-release-local',
    versionsToRemove: ['/3.2.0-build-'],
    dryRun: false]

Than we can ask for information about a folder. The following code block shows two functions, one to obtain information about a folder and the other to ask for a connection with the server.

def JSON folderInfo(path) {
    def binding = [repository: config.repository, path: path]
    def template = engine.createTemplate('''/artifactory/api/storage/$repository/$path''').make(binding)
    def query = template.toString()

    def server = obtainServerConnection()
    def resp = server.get(path: query)
    if (resp.status != 200) {
        println "ERROR: problem obtaining folder info: " + resp.status
        println query
        System.exit(-1)
    }
    return resp.data
}

private RESTClient obtainServerConnection() {
    def server = new RESTClient(config.server)
    server.parser.'application/vnd.org.jfrog.artifactory.storage.FolderInfo+json' = server.parser.'application/json'
    server.parser.'application/vnd.org.jfrog.artifactory.repositories.RepositoryDetailsList+json' = server.parser.'application/json'

    return server
}

We make use of the groovy SimpleTemplateEngine. As you can see we build the query string with placeholders and attach a map with parameters to the template engine. Next step is to generate the query. After obtaining the connection to the server we execute the GET request and obtain the JSON data from the response. Notice the lines in the obtainServerConnection. The last two lines add mappings for the artifactory provided content types. We want them to be treated as normal json responses or else our mapping framework (Jackson) will not be able to process the result.

Removing items

The goal for this blog post is to present a script to clean your artifactory repository. Now that we know how to obtain information about a folder in artifactory, it is time to have a look at cleaning. The idea is to take the provided path as a starting point. Go through all the folders recursively and check if we are in an artifact folder. An artifact folder contains artifacts and should not have other folders. If a folder is not an artifact folder we are dealing with a grouped (maven terms). In our case we could determine an artifact folder by its name. All artifacts have -build- in their name. Hence the following function.

private def isArtifactFolder(child) {
    child.uri.contains("-build-")
}

The next utility function is to actually delete a folder. The next function removes the provided path if we are not doing a dry run. If we did configure a dry run, we only print a message and do not actually do the delete. We again use the groovy template technology to create the query. Notice that the artifactory api uses the DELETE request.

private def removeItem(path, child) {
    println "folder: " + path + child.uri + " DELETE"
    def binding = [repository: config.repository, path: path + child.uri]
    def template = engine.createTemplate('''/artifactory/$repository/$path''').make(binding)
    def query = template.toString()
    if (!config.dryRun) {
        def server = new RESTClient(config.server)
        server.delete(path: query)
    }
}

The last piece of the puzzle is the recursive function to check a folder for subfolders and give the command to remove a folder if it is in the list of folder to be removed. The next code block shows this function.

def cleanArtifactsRecursive(path) {
    def deleteCounter = 0
    JSON json = folderInfo(path)
    json.children.each {child ->
        if (child.folder) {
            if (isArtifactFolder(child)) {
                config.versionsToRemove.each {toRemove ->
                    if (child.uri.startsWith(toRemove)) {
                        removeItem(path, child)
                        deleteCounter++
                    }
                }
            } else {
                if (!child.uri.contains("ro-scripts")) {
                    deleteCounter += cleanArtifactsRecursive(path + child.uri)
                }
            }
        }
    }
    return deleteCounter
}

The code is available online through github, check the references.

References

Cleaning up your maven repository

Other blogpost about groovy and cleaning your local maven repository automatically.

Artifactory

Homepage of the creators of Artifactory, here you can also find commercial support

Artifactory REST api

Documentation for the REST api that is provided by Artifactory

The source code

Sourcecode of the script discussed in this article.

http://blog.jteam.nl/2011/08/01/cleaning-up-your-maven-repository

If you want to find out more about this sample read on.

Introducing the sample

The sample is based on the famous address book from the axonframework project. We reuse the axon sample and create a rest like interface that returns json to the clients. The source code is in the rest-ui project within the axonframework source tree under samples. This project also contains the listener code for the Contact and Address based events. The listeners listen for events and publish content from these events to the redis pub/sub mechanism. More on this in the later sections.

The other part of the sample is a node.js application. This application makes use of the rest api to obtain the data via http requests. Events about data changes are received through redis, using the pub/sub mechanism. The node.js application is available on Github. If you want to follow along, you can find the sources here:

https://github.com/jettro/axon-addressbook-nodejs

http://code.google.com/p/axonframework/source/browse/trunk#trunk%2Fsample%2Faddressbook%2Frest-ui

The following screen dump gives you an idea about the application.

In the screen you see the list of contacts. You can add a new contact, change the name of a contact and add addresses to contacts. On the right you see the messages pane. Here you receive messages about new contacts that have been created, changed or deleted.

Now let us have a look at some of the code. We start with the java backend and then continue with the node.js part.

The java part of the sample

Numerous blogs have been written about rest based clients using spring mvc. Therefore I do not want to lose to much time in that area. Most important spring configuration is within the web.xml and the dispatcher-config.xml. In the web.xml we configure that every request is going to the spring dispatcher. We do not have status resources, only json responses. Within spring we make use of annotation to configure the beans. To be able to respond with json we make us of Jackson to transform beans into json objects. All pretty straightforward spring. Check the config file for yourself if you want to see how it works.

Next step is the controller that handles all the requests.

Handling requests in the Cntroller

The Controller handles the following requests:

• /contacts : GET, returns the list of all contacts
• /contacts : POST, creates a new contact
• /contacts : PUT, changes an existing contact
• /contacts : DELETE, removes a contact
• /contacts/{identifier} : GET, returns the details of a contact
• /contacts/{identifier}/address : PUT, creates a new address or updates an existing one based on address type
• /contacts/{identifier}/address : DELETE, removes the address of provided type

Let us have a look at one of the requests to show the flow. For example we take the request for details. The following code block shows the complete method.

@RequestMapping(value = "{identifier}", method = RequestMethod.GET)
public @ResponseBody Map<String, Object> details(@PathVariable String identifier) {
    List<AddressEntry> addressesForContact = repository.findAllAddressesForContact(identifier);
    String name;
    if (addressesForContact.size() > 0) {
        name = addressesForContact.get(0).getName();
    } else {
        name = repository.loadContactDetails(identifier).getName();
    }
    Map<String, Object> map = new HashMap<String, Object>();
    map.put("addresses", addressesForContact);
    map.put("identifier", identifier);
    map.put("name", name);
    return map;
}

The RequestMapping annotation tells spring which requests need to be mapped to this method. The controller already maps all /contacts to this controller. The {identifier} now maps everything with a url like /contacts/{identifier} to this method, as long as it is a GET request. This identifier is also used as a parameter for the method. With the @PathVariable we tell spring to take the value for this parameter from the url or Path that was requested. The final annotation in the method declaration is the @ResponseBody. This tells spring to use the return value as the response body, so not as the name of the view to use. We have already configured to use jackson to transform the response body into json. In the code of the controller you see we return a map with three objects in it. A list of addresses, an identifier of the contact and the name of the contact. The json coming out of this is:

{
"name":"Jettro",
"addresses":[
{
"name":"Jettro",
"identifier":"82b4fef5-3389-440d-8893-6831a64e200d",
"addressType":"PRIVATE",
"streetAndNumber":"Feyenoordlaan 010",
"zipCode":"3000AA",
"city":"Rotterdam"
}],
"identifier":"82b4fef5-3389-440d-8893-6831a64e200d"
}

That is what you need to know about the rest api that I have created. If you want to have a look at the controller, check here. The next step is about the listeners that listen for axon events related to the contacts and addresses and store information in redis.

Listening for events

We register listeners with axon for the contact related events as well as address related events. This is very easy with the annotations provided by the axonframework. The following code block shows how we register an event listener for the ContactCreatedEvent.

@EventHandler
public void handleContactCreatedEvent(ContactCreatedEvent event) {
    ContactEntry value = new ContactEntry();
    value.setName(event.getName());
    value.setIdentifier(event.getContactIdentifier());
    Message<ContactEntry> message = new Message<ContactEntry>("contact-created", value);
    publisher.publish(message);
}

Notice the @EventHandler annotation. This registers the listener with axon. As you can see we take the information from the provided event and put it in a Message object. The message object contains a type and the content. The type can be used at the receiving side to better understand how to handle the received event. The publisher is auto injected by spring. In my case it is a redis publisher.

Publishing to redis

A library to talk with redis is jedis. Jedis makes use of a pool which is configured in spring. Check the listener-context.xml to see how it works. Jedis does not have the most obvious api but it works. The following code block shows how a message is published to redis.

public void publish(Message<?> message) {
    StringWriter writer = new StringWriter();
    try {
        mapper.writeValue(writer, message);
    } catch (IOException e) {
        logger.warn("Problem while writing ContactEntry to a string writer", e);
        return;
    }

    Jedis jedis;
    try {
        jedis = jedisPool.getResource();
    } catch (JedisConnectionException e) {
        logger.debug("Could not obtain redis connection from the pool");
        return;
    }
    try {
        jedis.publish("nl.axonframework.examples.addressbook", writer.toString());
    } finally {
        jedisPool.returnResource(jedis);
    }
}

The code is not very complicated. The important part here is that we use a jackson provided ObjectMapper to map the provided message to a json representation. Then you can see the nice jedis api at work. As you can see we publish a message to a queue. Our node.js client will subscribe to this queue. That is have the redis publish/subscribe works. Not very hard is it?

That is it for the java side. We now know how to get data using the rest api and we know where to get information about events that take place. Let us move on to the node.js side of the application.

The Node.js part of the sample

I do not want to go in setting up a node.js project. Do not want to explain all the different libraries that I use. I have written other blog posts on node.js that I do not want to copy. The most important one is this:

http://blog.jteam.nl/2011/04/18/learning-node-js/

The node.js project uses 4 libraries:

• express – doing the mvc part of the web application
• jade – view templating
• now – server push to the client (will explain this in a bit more detail)
• redis – for redis interaction




Project layout

The public folder contains all the client related files like javascript, css and images. I will explain one file in here later on when discussing node.js. The test folder contains the test for the ContactRepository. Using this test helped me to verify that the integration between the java backend and the node.js application is working without having to start the application and go through the browser to check all requests. The views folder contains all the jade templates. Check mentioned blog post to find more information on jade. The app.’s initializes the application and maps all possible requests to methods in the ContactController. The controller knows where to get the data from and what page to render. The repository is responsible for obtaining data from the rest api on the java side.

Initializing the node.js application

As mentioned the app.js is the start of the application. Let us have a look at the different parts.

var ContactController = require('./ContactController');
var repository = require('./ContactRepository').createRepo('localhost',8080);
var contactController = new ContactController(repository);

This part initializes our components. We will have a look at the createRepo method later on. But notice that we initialize the controller with an instance of the repository.

app.get('/', contactController.listContacts);
app.get('/contact/new', contactController.newContactShowForm);
app.post('/contact/new', contactController.newContactPostForm);
app.get('/contact/:identifier',contactController.contact);
app.get('/contact/:identifier/edit',contactController.changeContactShowForm);
app.post('/contact/:identifier/edit',contactController.changeContactPostForm);
app.get('/contact/:identifier/delete',contactController.deleteContactShowForm);
app.post('/contact/:identifier/delete',contactController.deleteContactPostForm);
app.get('/contact/:identifier/address/new',contactController.newAddressShowForm);
app.post('/contact/:identifier/address/new',contactController.newAddressPostForm);
app.get('/contact/:identifier/address/:addressType/edit',contactController.changeAddressShowForm);
app.post('/contact/:identifier/address/:addressType/edit',contactController.changeAddressPostForm);
app.get('/contact/:identifier/address/:addressType/delete',contactController.deleteAddressShowForm);
app.post('/contact/:identifier/address/:addressType/delete',contactController.deleteAddressPostForm);

These are all the possible requests. The urls are easy to understand. As you can see, all urls are mapped to a method of the contactController

var Now = require('now');
var everyone = Now.initialize(app);

var redis = require("redis"),
    client = redis.createClient();

client.on("error", function (err) {
    console.log("Error %s", err);
});

client.on("message", function (channel, message) {
    everyone.now.receiveContact(message);
});
client.subscribe("nl.axonframework.examples.addressbook");

This is the harder bit. This has to do with receiving messages using the redis pub/sub mechanism and sending it to all connected clients using the nowjs library. I’ll discuss the now.js part later on.

Request handling

By using the repository object the controller becomes very easy. We only have to obtain the right data and render the right view. The most basic method is showing a list of contacts. The following code block shows the complete method.

ContactController.prototype.listContacts = function(req, res) {
    repository.listContacts(function(contacts) {
        res.render('index', {locals: {contacts: contacts, nowjs: true}});
    });
};

Notice that we use a callback that is provided to the repository listContact method. Check that we provide the list of contacts to the view, together with a second parameter called nowjs. Remember this, on this page we enable nowjs, meaning that we can receive messages when looking at this page. Other methods are similar, have a look at the method that handles submitting a form to create a new contact.

ContactController.prototype.newContactPostForm = function(req, res) {
    repository.newContact(req.body.new_name, function(code, message) {
        if (code == "ok") {
            res.redirect("/");
        } else {
            res.render('newcontact', {locals: {error:message}});
        }
    });
};

In line 2 we obtain the form parameter new_name from the request body. Than we create a new contact. The callback method expects a code indicating whether the submission of the new contact was ok. If not, we render the form again with the error message. This happens when we try to create a contact with a name that already exists. The following image shows the screen with an error message.

Next up is the communication with a rest based application

The Repository as a client to a rest application

During the initialization of the application I already mentioned the createRepo method of the ContactRepository. The following code block shows the method. As you can see it initializes a repository with the default values for host and port.

ContactRepository.createRepo = function(host, port) {
    var repo = new ContactRepository();
    repo.host = host;
    repo.port = port;
    return repo;
};

Again most of the interaction with the backend is the same. Therefore I use the create contact again as an example. I have created tests for all the repository methods. Let us have a look at the test first.

function testNewContact() {
    logTestName("New contact");
    repository.newContact("My Test", function(code, message) {
        assert.equal("ok", code, "This should be no problem and an ok should be returned: " + code);
        // Check if the amount of contacts is increased
        repository.listContacts(function(contacts) {
            assert.equal(numContacts + 1, contacts.length, "Number of contacts is not right, create did not work: " + contacts.length);
            testAddAddress();
        });
    });
}

Notice that we check the return code and that the amount of contacts is increased. The following code block shows the implementation of the method of the repository.

ContactRepository.prototype.newContact = function(name, callback) {
    var opts = createHttpRequestOpts('/contacts', 'POST');

    var req = http.request(opts, function(res) {
        res.setEncoding('utf8');
        res.on('data', function(data) {
            if (res.statusCode != 200) {
                console.log(data);
                callback('error', 'Maybe the name is already taken?');
            } else {
                callback('ok', 'The new contact has been send')
            }
        });
    });

    var contact = {};
    contact.name = name;

    req.write(JSON.stringify(contact));
    req.end();
};

function createHttpRequestOpts(path, method) {
    return {
        host: host,
        port: port,
        path:path,
        headers:{'Accept':'application/json','Content-Type':'application/json'},
        method: method
    };
}

I use the nodejs provided http request. Within the help function the opts object is created with the host, port, path, method and the headers. The headers are important to request json. Of course we provide a callback to handle the http response. Everything not 200 is an error as a response code. Here we call the provided callback to the repository method with the ok or error code. At the end of the method we do the actual request creation with the json request.

The final bit is the server push using the now.js library.

Server push using now.js

Node.js comes with a socket.io library. This makes server push very easy. Still it can be a lot easier. That is when using now.js. Now.js enables you to call methods on the client that are actually on the server and vise versa. Let us have a look at the code for sending messages to the client when a redis event is received again.

var Now = require('now');
var everyone = Now.initialize(app);
client.on("message", function (channel, message) {
    console.log("Received message: %s", message);
    everyone.now.receiveContact(message);
});

The important part here is the initialization of now and the line with everyone.now.receiveContact. Now we will call the connected clients method receiveContact and provide the message. Of course we need some client code for this. This code is available in the local.js file in the public folder.

$(document).ready(function() {
    now.receiveContact = function(data) {
        var message = JSON.parse(data);
        var toWrite;
        switch(message.type) {
            case 'contact-removed':
                toWrite = "Contact removed";
                break;
            case 'contact-created':
                toWrite = "Contact with name " + message.content.name + " created";
                break;
            case 'contact-changed':
                toWrite = "Contact changed name to " + message.content.name;
                break;
            case 'address-created':
                toWrite = "Added address of type " + message.content.addressType + " to contact " + message.content.name;
                break;
            case 'address-removed':
                toWrite = "Removed address of type " + message.content.addressType + " to contact " + message.content.contact.name;
                break;
        }
        $("#messages").prepend("<div>" + toWrite + " </div>");
    };
});

As you can see, we construct the message based on the type of the message and prepend a new div within the #messages div. Really that is all there is to it.

Last remarks

That is it. Again a long blog item. Hope you got some new knowledge out of it. Feel free to post comments if you have questions or even better improvements.

In this blog post I present you an additional sample, focussed on creating presentations as a programmer. I’ll use a few node.js modules: Jade, Express.

On my employers blog I wrote an extensive article about Node.js, you can have a look at the article as well.

http://blog.jteam.nl/2011/04/18/learning-node-js/

Requirements

I want to have a tool that supports multiple presentations that can all be shown in a browser. I do not consider a lot of browsers. I like the css support and the full screen view of chrome on the Mac. Therefore I test my work with Chrome only. I guess most of the features work on other browsers as well, some small issues might arise. A big advantage of using the browser is that you have lots of libraries available to create nice looking slides.

The tool needs to be developer friendly, therefore slides are created with code. No hot deploys, no tools to just enter slides. The slides are within the code. This is not an end user tool, this is a developer tool.

Each presentation consists of a title, should be able to work with a chosen template and contains slides of course.

Giving or presenting the presentation should be easy as well. Therefore I want to be able to choose a slide, go to the next slide, the previous slide and the index of all presentations.

Going to a different presentation means using another url. For each presentation we provide a path and the slide index. A url for a presentation skills presentation than becomes:

http://dev.gridshore.nl:8018/presentation/create-presentation-nodejs

The first slide in that presentation than becomes:

http://dev.gridshore.nl:8018/presentation/create-presentation-nodejs/slide/1

The final feature that would be nice is generating a handout. For now this will be a print view in the browser, maybe a pdf. An overview with all the slides on one page will be here:

http://dev.gridshore.nl:8018/presentation/create-presentation-nodejs/all

Chosen technologies

A slide consists of content, layout and styling. I want to limit the amount of styling information in the content. Still I want to make it simple. Therefore I have chosen to use a template engine for the front-end. I have some experience with Jade, which seems to do the job fine. The styling is done using css and where necessary, with additional components. An example is presenting source code on a slide.

When working with a web application and a browser it is wise to have a way to handle requests. Express.js is a nice node.js framework to use for url handling. I’ll get back with some sample code later on.

Since we are creating a web application, we need style sheets and images. I also use jQuery and a few plugins to position elements, to handle keyboard commands and to show source code. Please refer to the references section if you need more information about one of the technologies.

The solution

Project layout

An express.js application consists of an app.js, a public folder and a views folder. The app.js is the start of the application that you call with the node executable. In the public folder we have the static web resources. It contains the stylesheets, client-side javascripts and other static resources. The views folder contains the layout.jade file and all the jade views. Within the images and the views folder we have a subfolder for each presentation. I prefer to use the same path as the path specified for the url for the presentation. More on this later.

The project now has two dependencies: expressjs and jade. I like to manage my dependencies with npm.

Creating the presentation

The presentation is created as a separate component, have a look at presentation.js. It contains the title of the presentation, the urlIdentifier, a description and the slides. The urlIdentifier is used in the url when requesting the presentation in a browser. The slides are also components. You can find the slide component in slide.js. The slide contains an identifier and a title. The identifier is important, this is used to find the right jade view file. The following code block shows the creation of the sample presentation.

var createPresentationNodejs = [];
createPresentationNodejs.push(new Slide("intro","Creating a presentation with Node.js"));
createPresentationNodejs.push(new Slide("goal","Goal"));
createPresentationNodejs.push(new Slide("requirements","Requirements"));
createPresentationNodejs.push(new Slide("technology","technology"));
createPresentationNodejs.push(new Slide("urlhandling","Configure url handling"));
createPresentationNodejs.push(new Slide("handlerequests","Handle requests"));
createPresentationNodejs.push(new Slide("viewcontent","View content"));
createPresentationNodejs.push(new Slide("thefuture","The Future"));
createPresentationNodejs.push(new Slide("questions","Questions"));

presentationController.addPresentation(new Presentation("Creating a presentation with Node.js","create-presentation-nodejs",
	createPresentationNodejs,
    "For my work I had to prepare a short presentation of around 10 minutes. I decided to create a tool to create presentation " +
            "with and present about it. This is the result, a short presentation showing some of the ideas."));

In the first lines you can see how we create the array of slides. The last few lines show how we add a new presentation to the presentationController. We get back to this presentationController later on. First we have to look at express and url handling.

Express and url handling

Express is a web framework for node.js. You can do lots of things, in this blog post I do not want to give to much basics. Check the blog post I wrote on my employers blog: http://blog.jteam.nl/2011/04/18/learning-node-js.

THe thing I do want to mention is the url mapping for GET and POST requests. These are important for the flow of the application. You can also see the way that the PresentationController is used. Have a look at the following code block

var presentationController = new PresentationController();

app.get('/', presentationController.allPresentations);
app.get('/presentation/:urlIdentifier', presentationController.index);
app.get('/presentation/:urlIdentifier/slide/:id', presentationController.slide);
app.post('/presentation/:urlIdentifier/slide/:id', presentationController.command);
app.get('/presentation/:urlIdentifier/all', presentationController.allSlides);

Going to the root of the application gives a list over presentations. Asking the presentation with the urlIdentifier of the presentation gives an overview of the presentation. You can also ask the /all, this will give all the slides. Check the section about a slide handout for more information on this.

The handling of the POST request is for navigation. Within the slide you can enter n for next, p for previous, a number for a specific slide and enter for the next slide. This will result in a command on the server using the post request. Creating this request can be found in the layout.jade. It is a normal form for the action to the url of the current slide number. Using the command and the current slide number we can determine the slide to select. The next block shows the client code, handling the post request is discussed in the next section.

form(action="/presentation/"+urlIdentifier+"/slide/"+numSlide,method="POST")
    input(type="text", size="100", name="command", id="command")

Now we have seen the url handling, let us move on to the logic in the PresentationController

The presentation controller

The presentation controller contains the logic to handle a request and send the response back to the requestor. The following code block shows the the handling for the request to show all available presentations.

PresentationController.prototype.allPresentations = function(req, res) {
    res.render('index', {locals: {presentations:presentations}});
};

The presentations is an internal array for the presentation controller. They are passed to the ‘index’ view. More on the view with jade in the next section. The following code block is slightly more advanced. It contains the logic to show one slide.

PresentationController.prototype.slide = function(req, res) {
    var id = req.params.id;
    var presentation = obtainPresentation(req.params.urlIdentifier);
    var slide = presentation.slides[id - 1];
    res.render(presentation.urlIdentifier + '/slide/' + slide.identifier,
            {locals: {
                numSlide:req.params.id,
                totalSlide:presentation.slides.length,
                titleSlide:slide.title,
                urlIdentifier:presentation.urlIdentifier
            }});
};

function obtainPresentation(urlIdentifier) {
    var len = presentations.length;
    for (var i = 0; i < len; i++) {
        var presentation = presentations[i];
        if (urlIdentifier == presentation.urlIdentifier) {
            return presentation;
        }
    }
    throw "Could not find the url: " + urlIdentifier;
}

The code contains a helper function to determine the presentation to show a slide for. The presentation is obtained by the url identifier. The page is rendered by using the specific jade template that is stored using the urlIdentifier in the slides folder. The name of the jade template is the same as the identifier as stored in the slide object.

The last bit I want to discuss is the command handling. We have a small form at the bottom of the page. This form accepts commands. We support ‘n’ or ‘next’, ‘enter’, ‘p’ or ‘previous’ and the number of a slide. These all navigate between the slides. We also have the ‘home’ command. This redirects to the root of the application. Handling the different commands is done with a very easy if-the-else structure. The following code block shows the idea.

PresentationController.prototype.command = function(req, res) {
    var id = req.params.id;
    var urlIdentifier = req.params.urlIdentifier;
    var command = req.body.command;
    var presentation = obtainPresentation(urlIdentifier);
    var numSlides = presentation.slides.length;

    if (command == 'next' || command == 'n' || command == '') {
        if (id < numSlides) id++;
    } else if (command == 'previous' || command == 'p' || command == 'prev') {
        if (id > 1) id--;
    } else if (command == 'home') {
        res.redirect('/');
        return;
    } else if (!isNaN(command)) {
        var commandNum = parseInt(command);
        if (commandNum > 0 && commandNum &lt;= numSlides) id = commandNum;
    }

    res.redirect('/presentation/' + urlIdentifier + '/slide/' + id);
};

Enough about logic, let us move on to the view side and discuss templating using Jade.

Little bit of jade

Again I do not want to go into all the details of jade. Jade is a very condensed format for creating html pages. It integrates nicely with express. Data is passed from express to jade. Jade uses a layout file, you can also use partial rendering without a layout file. More on this in a next section. In this case we include the header and the footer in the layout file. Each slide is presented in its own jade template file. The following code block shows the jade template for the intro slide in the sample.

img(src='/images/' + urlIdentifier + '/logo.jpg')
div.heading author
div.heading Jettro Coenradie
img(src='/images/' + urlIdentifier + '/nature.jpg', class="shadow")

Another nice thing about using the web browser as a tool is that you can use nice plugins available for websites to create your slides. A good example is showing source code. This is what the next section is about.

Showing code

To show source code I use a syntax highlighter from alex gorbatchev. The client side library files are included in the public folder. Showing code in a jade template looks like this.

pre(class="brush: js")
    | app.get('/', function(req, res) {
    |     res.render('index',
    |         {locals: {slides:slide.allSlides()}});
    | });
    | app.get('/slide/:id', slide.index);
    | app.post('/slide/:id', slide.command);
script(type="text/JavaScript")
    | SyntaxHighlighter.all()

Creating a slide handout

One of the requirements of a presentation is often to have the slides available as a pdf document or other format that is able to share the slides. The path I am taking is using one page for a presentation that contains the complete presentation. I would love to have a pdf generated from the webpage. But for now this is one step to far. I did have a look at using node.js child processes to run a command line tool called wkhtmltopdf. I want to do some more experimentation here. Will get back when I have a good solution. Beneath are some references if you want to try it out yourself.

• http://blog.darkhax.com/2010/11/29/2-node-js-apps-that-showed-me-the-light
• http://code.google.com/p/wkhtmltopdf/




To be able to create one page with all the slides I made use of the partial rendering of jade and express. The idea is to pass a the presentation to a view. The view steps over all the slides in the presentation and renders the views on one page. The following code block shows the view (all.jade).

div#All slides
h2 #{presentation.title}
ul
    - each slide in presentation.slides
        - var slideTemplate = presentation.urlIdentifier + '/slide/' + slide.identifier
        .slideAll
            - if(typeof(slide.title) != "undefined")
                div.title #{slide.title}
            !=partial(slideTemplate, {'urlIdentifier':presentation.urlIdentifier})

The “presentation” object is provided by the express component. From this object we obtain the path in the views folder. The presentation object also contains the slides. Each slide has its own identifier. Using the slide identifier and the urlIdentifier we can find the jade template to render. The trick is in the last line !=partial(…). This tells the engine to include the specific slide templates in he current view.

The demo

There is an online demo available at: http://dev.gridshore.nl:8018/. It contains only one presentation at the moment. But that is enough to get an idea. This blogpost also contains two screen dumps.

The future

I want to integrate the pdf generation within the application. I have some hints, but not the complete solution. Maybe extend the tool in a way that you can create new presentations on the fly, but I do not want to make an alternative to presentation tools that are already there. I also want to add more controls, would be nice to be able to use the arrow keys and maybe support remote controls that way.

References

• http://nodejs.org/ – Node.js home page
• http://expressjs.com/- Express.js
• http://jade-lang.com/- Jade
• http://alexgorbatchev.com/SyntaxHighlighter- Code beautifier
• https://github.com/jettro/nodejs-presenter – the code for the sample.
• http://blog.darkhax.com/2010/11/29/2-node-js-apps-that-showed-me-the-light – inspiration for pdf export.
• http://code.google.com/p/wkhtmltopdf/ – want to use for pdf export.