In my previous blog posts I was working on reading data from wordpress blogs using groovy. This is nice, but of course there was a reason why I needed this. I wanted to create a tool or better a plugin for Elasticsearch. This plugin should make it easier to check the state of you cluster, play around with facets and query your data.

On my employers blog I started a series of blog posts that explain this plugin. I go into details for the libraries I used: AngularJS, Twitter Bootstrap and of course elastic.js.

Check my employers blog post if you are interested.

Introducing a query tool as an elasticsearch plugin (part 1)/

The post Introducing a query tool for elasticsearch appeared first on Gridshore.

But what if you want to start playing with things like facets? What if we want to use a different analyzer to separate the keywords on a comma? Than you can use curl. No wait there is more. You can also use groovy of course.

That is what I discuss in this blog post, creating and removing indexes. Beware, this is not something you want to do on your production server. Deleting an index makes the index disappear, yes for real, you cannot get it back.

Read on if you want to learn more about groovy and ElasticSearch.

Prequel

Please read my previous post if you want to understand the sample that I am creating. I will not discuss it in this blog post.

Learning about ElasticSearch/

Delete an index

Again be careful, usually this is not possible in production. You should disable it.

The ElasticSearchGateway contains the following method to delete an index.

public deleteIndex() {
    try {
        def response = node.client.admin.indices.prepareDelete(indexValue).execute().get()
        if (response.acknowledged) {
            println "The index is removed"
        } else {
            println "The index could not be removed"
        }
    } catch (Exception e) {
        println "The index you want to delete is missing : ${e.message}"
    }
}

node is an instance of GNode created in the constructor of my Gateway class. You ask for the client to get access to the ElasticSearch cluster. Than ask for the admin part on which you can call index related queries. In this case we call the prepareDelete method. I put a try-catch around the delete statement. I do not want the script to stop if the index cannot be deleted because it is not there.

Creating the index

Before I show you have to create an index with some advanced stuff going on in there, I want to explain why I needed it. I want to create facets around the keywords and the categories in my blogs. The format of the keywords field is:

Yvonne van der Mey, fotografie, bloemen

I want three terms out of this after analyzing for creating facets. Therefore I want to tokenize based on the comma and I want to strip the spaces from the items. This can be done with a custom analyzer. The custom analyzer refers to a pattern based tokenizer and a trimming filter. In groovy you can create this index, with settings and mappings as shown in the next block.

public createIndex() {
    def future = node.client.admin.indices.create {
        index = this.indexValue
        settings {
            number_of_shards = "1"
            analysis {
                analyzer {
                    comma {
                        type = "custome"
                        tokenizer = "bycomma"
                        filter = ["nowhite"]
                    }
                }
                tokenizer {
                    bycomma {
                        type = "pattern"
                        pattern = ","
                    }
                }
                filter {
                    nowhite {
                        type = "trim"
                    }
                }
            }
        }
        mapping this.typeValue, {
            "${this.typeValue}" {
                properties {
                    keywords {
                        type = "string"
                        analyzer = "comma"
                    }
                    categories {
                        type = "string"
                        analyzer = "comma"
                    }
                }
            }
        }
    }

    future.success = { CreateIndexResponse response ->
        println "Index is created"
    }

    future.failure = {
        println "ERROR creating index $it"
    }

}

With groovy you can use the closure notation to create these items like settings and mappings. This way the method with a string containing the source is called and the closure is transformed into the json structure. The notation looks very similar to the json as provided when using curl.

A nice way to test if your analyzer works is using curl. You can actually call the analyzer with a string to be analyzed.

[~]$ curl -XGET 'localhost:9200/coenradie/_analyze?analyzer=comma&pretty=true' -d 'this ,is a ,test, with a lot of,difference,'
{
  "tokens" : [ {
    "token" : "this",
    "start_offset" : 0,
    "end_offset" : 5,
    "type" : "word",
    "position" : 1
  }, {
    "token" : "is a",
    "start_offset" : 6,
    "end_offset" : 11,
    "type" : "word",
    "position" : 2
  }, {
    "token" : "test",
    "start_offset" : 12,
    "end_offset" : 16,
    "type" : "word",
    "position" : 3
  }, {
    "token" : "with a lot of",
    "start_offset" : 17,
    "end_offset" : 31,
    "type" : "word",
    "position" : 4
  }, {
    "token" : "difference",
    "start_offset" : 32,
    "end_offset" : 42,
    "type" : "word",
    "position" : 5
  } ]
}

As you can see, the spaces at the end and beginning are removed, but we have multiple words as one token. Exactly like I want.

The code also shows we use this analyzer for two fields: keywords and categories. The next screen shows another project I am working on. This project is actually the main project but I needed some content so I created the groovy scripts to find content and send them to ElasticSearch. Check the facets at the bottom of the screen.

I hop you like this addition. My next blog will be about ElasticSearch and writing a plugin using AngularJS.

The post Doing more ElasticSearch with groovy appeared first on Gridshore.

This week I had a training at Trifork Amsterdam by Martijn and Uri from ElasticSearch. This training was a very nice in depth look at the capabilities of ElasticSearch. Like with all trainings and conferences I get motivated to try out the technology immediately. I am working on a plugin for ElasticSearch using AngularJS. More on this in a next blog at Trifork.

In this blog I am going to tell you the steps I took to get going with ElasticSearch. I show some of the steps for installing ElasticSearch, explain the first steps in using the groovy library. Next to that I’ll show you how to index this wordpress blog using ElasticSearch. When everything is indexed I will of course show some of the queries you can perform.

Let us get going.

Installing and configuring

I do not want to make this part to extensive. The documentation at the site of ElasticSearch is rigorous and therefore my extra lines would be to much. Some of the things I want to accomplish in the configuration are:

• Start elastic search with the logs in your command prompt: -f
• Specify the location of the configuration file: -Des.config=/path/to/config/file
• Change the name of the cluster and the node: cluster.name: jc-elasticsearch and node.name: “Node-gridshore”
• Disable the multiple shards and replicas for local development: index.number_of_shards: 1 and index.number_of_replicas: 0
• Path to the location of the configuration files: path.conf: /path/to/conf
• Other paths, like work folder, logs folder, data folder and plugin folder.
• Install the head plugin to have a look at whay is happening to your cluster. I have chosen to install the plugin in the normal directory and than copy it to my configured plugins folder.

Don’t forget to install and use java 7, al that java 6 stuff is really old and slow compared to 7. ElasticSearch can do a better job when running on 7.

Those are the first commands to get everything running, so once more the steps:

1. Download elastic search: http://www.elasticsearch.org/download/. I used 0.20.2
2. Create a directory structure outside of the downloaded elastic search folder: config, data, logs, plugins, work.
3. Copy the elastic search.yml and logging.yml to the config folder and make the changes as mentioned.
4. Install the plugin and copy it to your plugins folder: bin/plugin -install Aconex/elasticsearch-head
5. Move to the main folder of you ElasticSearch download and execute the command (with your location of the config file)

bin/elasticsearch -f -Des.config=/.../elasticsearch/projects/gridshore/config/elasticsearch.yml

Now you can browse to the head plugin and see that not a lot has happened yet.

http://localhost:9200/_plugin/head/index.html

Before going to the index creation you can play around with some aspects of the REST api that ElasticSearch exposes: Try a few of these requests with curl to see the data that ElasticSearch exposes. Don’t use the pretty=true in production.

http://localhost:9200/_cluster/state?pretty=true

http://localhost:9200/_cluster/nodes?pretty=true

Reading the data using Groovy

Of course you can create the index and the required mapping using the curl based api. You can also just insert a document and the index will be created automatically and the default mapping will be created. In our case I first want to introduce you to the groovy client I use to read data from the blog using xmlrpc and the data model. Than we move on to the mapping.

In the sample code I use gradle to configure the project and my environment (Intellij).

dependencies {
	groovy 'org.codehaus.groovy:groovy:2.0.6'
    groovy 'org.codehaus.groovy:groovy-all:2.0.6'
    groovy ('org.codehaus.groovy:groovy-xmlrpc:0.8') {
        exclude module: 'groovy-all'
    }
    groovy 'commons-cli:commons-cli:1.2'
    groovy 'log4j:log4j:1.2.16'

    groovy ('org.elasticsearch:elasticsearch-lang-groovy:1.2.0') {
        exclude module: 'groovy-all'
    }
}

There is some strange thing going on with the groovy and groovy-all libraries. But I had some issues combining intellij and gradle. This seems to work for me. Notice that we need the groovy-xmlrpc library and of course the elasticsearch-lang-groovy library.

Using the following groovy class we can read all posts from the gridshore blog. Since the code is straightforward, I am not going in any details.

package nl.gridshore.wordpress

import groovy.net.xmlrpc.XMLRPCServerProxy

class WordPressReader {
    private String xmlrpcUrl
    private String username
    private String password

    private XMLRPCServerProxy serverProxy;

    def WordPressReader(xmlrpcUrl, username, password) {
        this.xmlrpcUrl = xmlrpcUrl
        this.username = username
        this.password = password

        serverProxy = new XMLRPCServerProxy(xmlrpcUrl)
        serverProxy.setBasicAuth(username, password)
    }

    def obtainMostRecentPosts(int number = 10) {
        def posts = []
        def foundPosts = serverProxy.metaWeblog.getRecentPosts(1, username, password, number)
        foundPosts.each {post ->
            def blogItem = new BlogItem()
            blogItem.id = post['postid']
            blogItem.link = post['permaLink']
            blogItem.status = post['post_status']
            blogItem.keywords = post['mt_keywords']
            blogItem.title = post['title']
            blogItem.createdOn = post['dateCreated']
            blogItem.content = post['description']
            blogItem.categories = post['categories']
            blogItem.author = post['wp_author_display_name']
            blogItem.slug = post['wp_slug']
            posts.add(blogItem)
        }
        return posts
    }
}

Time to put some stuff into ElasticSearch

Put some data into the index

The groovy library is very easy to use. The following code block shows the opening and closing of the connection. We make use of a client node that does not contain data. Take special notice of the line where we set a property for GXContentBuilder, this is required to enable the configuration using a closure. In the constructor we open the connection and in you have to close the connection using the close method.

package nl.gridshore.elasticsearch

import nl.gridshore.wordpress.BlogItem
import org.elasticsearch.action.index.IndexResponse
import org.elasticsearch.groovy.common.xcontent.GXContentBuilder
import org.elasticsearch.groovy.node.GNode
import org.elasticsearch.groovy.node.GNodeBuilder

import static org.elasticsearch.groovy.node.GNodeBuilder.nodeBuilder

class ElasticSearchGateway {
    GNode node

    ElasticSearchGateway() {
        GXContentBuilder.rootResolveStrategy = Closure.DELEGATE_FIRST; // required to use a closure as settings

        GNodeBuilder nodeBuilder = nodeBuilder();
        nodeBuilder.settings {
            node {
                client = true
            }

            cluster {
                name = "jc-elasticsearch"
            }
        }

        node = nodeBuilder.node()
    }

    public close() {
        node.stop().close()
    }
}

Now that we know how to obtain a connection, let us obtain some data and store the documents in the ElasticSearch index. The next method is also coming from the ElasticSearchGateway class that I have created. This is the most basic version that auto creates and index and the complete mapping.

    public indexBlogItem(BlogItem blogItem) {
        def future = node.client.index {
            index = "gridshore"
            type = "blog"
            source {
                blogId = blogItem.id
                link = blogItem.link
                status = blogItem.status
                keywords = blogItem.keywords
                title = blogItem.title
                createdOn_date = blogItem.createdOn
                content = blogItem.content
                categories = blogItem.categories
                author = blogItem.author
                slug = blogItem.slug
            }
        }

        future.success = {IndexResponse response ->
            println "Indexed $response.index/$response.type/$response.id"
        }
    }

With the following script we obtain 100 items from my blog and add them to the ElasticSearch index.

import nl.gridshore.elasticsearch.ElasticSearchGateway
import nl.gridshore.wordpress.BlogItem
import nl.gridshore.wordpress.WordpressReader

def rpcUrl = "http://www.gridshore.nl/xmlrpc.php"
def username = "?"
def password = "?"

def reader = new WordPressReader(rpcUrl,username,password)

def posts = reader.obtainMostRecentPosts(100)

ElasticSearchGateway gateway = new ElasticSearchGateway()

posts.each {BlogItem item ->
    println item.title
    gateway.indexBlogItem(item)
}

System.in.withReader {
    print 'input: '
    println it.readLine()
}

gateway.close()

Execute queries on the data

Now that I told you I have data in the index, let us create a query to check if there is actually something in the index. Of course we use groovy to query the index and to print the results.

    public queryIndex(theTerm) {
        def search = node.client.search {
            indices : "gridshore"
            types : "blog"
            source {
                query {
                    term(_all: theTerm)
                }
            }
        }

        search.response.hits.each {SearchHit hit ->
            println "Got hit $hit.id from $hit.index/$hit.type with title $hit.source.title"
        }
    }

By running the following script I get the shown results.

import nl.gridshore.elasticsearch.ElasticSearchGateway

ElasticSearchGateway gateway = new ElasticSearchGateway()

gateway.queryIndex("groovy")

System.in.withReader {
    print 'input: '
    println it.readLine()
}

gateway.close()

Got hit Huib3BPiTEua7Yvy6kc83w from gridshore/blog with title Doing more with groovy
Got hit lp79gd1UQ6mSdp3yp7O8Jg from gridshore/blog with title Analyzing beet results with groovy
Got hit CoRdYwD9ScyNnocqDvLGYg from gridshore/blog with title Cleaning up your maven repository with groovy
Got hit _U2q3miSRn6dTdUJB9PNYA from gridshore/blog with title Cleaning up artifactory with a groovy script
Got hit StrcQtnwTJSIvlu0pYojpA from gridshore/blog with title Exposing jmx through jmxmp and reading the jmx data with groovy
Got hit iQz0QFtYT3af9-Rls4d8Zw from gridshore/blog with title Use Grails and Axon to create a CQRS application (part II)
Got hit fajgAgrgST6AnNwCeyyocA from gridshore/blog with title Recap of the year 2010
Got hit V6h65a_rQ2mOvpE-POTF4Q from gridshore/blog with title Use Grails and Axon to create a CQRS application (part I)
Got hit xdi2KN9zQf2JD8qdbhBmdQ from gridshore/blog with title Using the NOS open data API with the springframework and jackson
Got hit DOiahRyMS0KUbg9e6BQHpQ from gridshore/blog with title Doing grails, yes I like it

The last function I want to show is counting documents. The following code block shows how to count all documents in the gridshore index of type blog. Thinking about the input for the amount of blogitems we imported, the result should be obvious.

    public countAllDocuments() {
        def count = node.client.count {
            indices : "gridshore"
            parameterTypes : "blog"
        }
        
        println "Number of found blog items : $count.response.count"
    }

Number of found blog items : 100

This is it, you can find the sources online @github. Check my GridshoreSamples project, in there you’ll find a small project called groovy-es-client.

https://github.com/jettro/GridshoreSamples

The post Learning about ElasticSearch appeared first on Gridshore.

In the past I have read multiple books about creating and giving presentations. I really like to present something to the public. On numerous NLJug events I have presented as well as a lot of internal events @ trifork.

These books have helped me to improve my presentation skills as well as the process to come up with nice presentations.

If you are also interested in these kind of books and you do not mind that I earn a few bucks you can buy these books that I highly recommend @ Amazon.

The post Review of the book: Presentation Patterns appeared first on Gridshore.

Twitter

I am not a very active user on twitter with @gridshore. I do twitter more with my personal account @jettroCoenradie. Still it is not hard to see what I was busy with this year. Check my twitter overview created with vizify

Presentations

This year I presented at two events. At the Hippo Gettogether I presented about the ldap integration we did at the University of Amsterdam. You can find the slides here at slideshare.

At the yearly JFall conference of the NLJug i presented about creating polyglot and scalable applications. You can download the slides, but you can also watch the presentation on Parleys.

Blogs

When looking back I see there is not a lot of activity on the gridshore website. Most of my activity moved to my employers blog. Therefore I have decided to write a short post in here as well when I write a blog post at my employers blog.

My year started with a blog post about running the hippo cmd components from within intellij. At Hippo they preferred the cargo plugin to run their components, which is not ideal when debugging your software. Therefore I came up with a way to make it easy to run your components in intellij.

The next blogpost I wrote was about creating a MongoDB based event store for the Axonframework. Allard made some nice improvements to the event store, but this was the foundation. The event store is used in a sample application called the Axon Trader. More on this later on.

I am a hobbyist photographer. I attend a number of workshops every year at Yvonne van der Mey. She needed a new website and I like to help people out. Therefore I created a wordpress based website for her. I have a number of other wordpress website under maintenance (www.wateenjuweeltje.nl, www.coenradie.com,www.nicobulder.com). In the beginning I was doing my editing on the server, but I wanted to have a better solution. I started using git to have a history for my scripts. Manually copying the sources to the server was not nice. Therefore I created a few scripts to check git what changes were not send to the server yet. Using ftp I send these files to the server. Of course I wrote a blog post about this: Deployments with Git and Bash on a Mac.

I continued building applications using Axonframework and MongoDB. I already created the event store, but an application using axon also has a query side. For the Axon Trader sample I wanted to use MongoDB for the query side as well. I heard about the spring-data project and thought there was a good match for my sample. In this blog post I discussed my experiences with the spring-data project and the MongoDB module specifically.

For a number of projects we wanted information about the server using a ping request. I thought it was good to create something reusable, therefore I created the Healthcheck library. In this blogpost you can read more about the java based health check library.

For the University of Amsterdam we are creating a Hippo based solution for their new websites. Like with other hippo projects we have a lot of integration to do. I usually create an administrator application for managing all these integration components. Of course we do not want everybody to be able to login, therefore we have created an authentication mechanism for Spring-security to authenticate against a hippo repository. This blog post discusses that mechanism.

During 2012 I started learning about Vert.x. This is a very nice application framework to create scalable/polyglot applications on the JVM. I had an idea about a presentation proposal for the NLJuG JFall event. Before being able to present about it I had to learn. Usually I learn by creating sample applications and blogging about my experiences. Therefore I wrote this blog post about my first steps with vert.x. This was a very well read blogpost that was tweeted about 32 times. Nice to know that people liked what I was doing.

During the Devoxx conference I learned about AngularJS. I really liked the idea around this JavaScript framework. It also seemed easy enough to work with. With my interest in Vert.x and Axonframework I wanted to combine this knowledge in a sample application. This blog post described my experiences: Basic Axon Framework sample using vert.x and angular.js.

Conferences

In May I attended the GoTo conference in Amsterdam. At Trifork we organise all the GoTo conferences. The one in Amsterdam was a nice conference this year. I learned a lot and got really inspired by a few talks.

In November I was able to join the Devoxx conference. What I thrill this is. I attended a lot of presentations and we joined a booth with 10gen, the creators of MongoDB. I had some very interesting talks that made me think and want to try out new stuff. I really liked the talk about AngularJS. The evening I got back I rewrote a part of a sample using angularJS. This is really something for the future. Here you can read more about my devoxx 2012.

iOS

In this year I also started learning about iOS development. I started with an app for the iPad. But since both my kids have an iPod touch I wanted to create a very basic application to learn mathematics for the kids. I’ll write more about this in 2013.

Books

Like every year I spend some money on books. A lot of them are eBooks, but I also keep buying hard copy books.

• Presentation patterns – Neal Ford
• The lean startup – Eric Ries
• Just enough software architecture – George Fairbanks
• Java Performance – Charlie Hunt
• Don’t make me think – Steve Krug
• Spring Integration in Action – Mark Fisher
• 7 databases in 7 weeks – Eric Redmond
• iOS Storyboarding – Dr. Rory Lewis
• The iOS 5 developers Cookbook – Erica Sadun
• iOS SDK Development – Chris Adamson

The post This was my 2012 appeared first on Gridshore.

Both my sons play soccer. I am one of those fathers that played soccer themselves and therefore think they are qualified to be a soccer trainer. Well, someone needs to do it and the kids and parent tell me I do a good job. Of course I did some research on the internet, watched youtube movies and I even attended a “train the trainer training” (nice sentence). When training the kids (around 10 of them) I ask them to try certain exercises like tip the ball with your toe, pass the ball to someone else and wait to get it back. I think you get the idea. During the training I try to combine hard work with having fun. Still the first thing the kids ask is: when are we going to play a match? However, when explaining a new exercise, I am glad they do not ask “Why should I want that?”.

At work I am one of the seniors of the company. I like to mentor/coach my colleagues. I think it is great to see interns grow from still in school “I know the world guys” into men who have actually been part of something. I like to think about the future together with them, and make plans. Talk about what they want to become and the steps they need to take. I discuss the questions they need to ask themselves. I think it is very important to like what you are doing at work. It is hard to keep up if you are not having fun. One thing that surprises me is that colleagues very often forget to ask the question “Why should I want that?”

The post Why should I want that? appeared first on Gridshore.

For the axon trader sample I am working on I want to have a screen of executed trades. I want the screen to be updated as soon as possible. I have some wild ideas of connecting the vert.x event bus to the axon event bus, but that is not the focus for this blog post. In this blog post I am going to step through the stages of creating a sample vert.x application that makes use of sockjs to visit a page where you can connect to the event system and get updates about the executed trades. The executed trades are stored in a MongoDB instance. I use the query tables from the axon-trader sample, but this is not a requirement for you to replay the sample.

I am also not going to explain what vert.x is, the website http://vertx.io/ does a good job already. Expect a lot of code samples and of course the source is available in github. Check the references at the end.

If you are still interested, the click the read more link and start reading. Feedback and improvements are appreciated.

Create a web app

We start of creating a very basic web application. No static files, no ajax, no web2.0. Just a request and a response. The following code block shows you how to create a web server serving three urls: /trades, /orders/buy and /orders/sell. I also wanted to create a request nog found page, but this was not working in my current code base using groovy.

def server = vertx.createHttpServer()
def routeMatcher = new RouteMatcher()
routeMatcher.get("/trades") {req ->
    req.response.end "You requested trades"
}
routeMatcher.get("/orders/:type") {req ->
    def type = req.params["type"]
    if (type == "sell")
        req.response.end "You requested buy orders"
    else if (type == "buy")
        req.response.end "You requested sell orders"
    else
        req.response.end "You requested orders of unknown type"
}
server.requestHandler(routeMatcher.asClosure()).listen(8080, "localhost")

As you can see we need to create the http server and a Routematcher. Then we specify the routes to match. The trades is very easy and the orders are a little bit more complicated using a query parameter, but you should be able to understand this as well. The final registers the handler for the event of receiving a request and initializes the listener using the server name and the port.

Next step is obtaining data from mongoDB using the provided mongodb BusMod.

Attach the web app to mongodb

In the nature of vert.x and using events, we use the event bus to send a message to mongo and receive the reply in a callback. But before we can send a message, we need to initialize the mongo verticle. The next code block shows setting up the mongodb vertical using the defaults for address,host and port. The address is interesting, this is the address in the event bus that the mongodb persistence verticle listens for incoming requests. This default is vertx.mongopersistor. As you can see we use the mongo database for the axon trader. See references for more information about this sample.

def mongoConfig = ["db_name":"axontrader"]
container.with {
    deployVerticle('mongo-persistor', mongoConfig)
}

Before we can send messages to the event bus, we need to have a reference to the event bus. This is easy to obtain.

def eventBus = vertx.eventBus

Now we can send messages. I already mentioned the address to send the messages to. When sending a message to obtain data you need to provide the collection to read the data from and a matcher to filter the items that you need. I had some problems finding the right format for the query (see the problems section). You also want to provide a callback that receives the data that was found. The next code block shows how we return data back to the client that was found in mongo.

routeMatcher.get("/trades") {req ->
    def query = ["action":"find","collection":"tradeExecutedEntry","matcher":[:]]
    eventBus.send("vertx.mongopersistor",query) {message ->
        req.response.end "The following trades were found: ${message.body}"
    }
}

Have a look at the query where we specify the action, the collection and the matcher. There are other options, check the manual for them. The result is a message. The message contains the found data in the body. Also notice the used address for sending the message to using the event bus.

Since we want a human readable website, we cannot show a string containing json. Therefore we also want to have a better looking front end.

Create a client that shows the data

Creating a nice website involves providing access to some static files. Think about style sheets, javascripts and images. Of course vert.x has support for this. Vert.x comes with a verticle for serving static content. This is the WebServer. A nice functionality of this module is that you can use a bridge. This way the clients can send requests to the event bus immediately. A nice functionality, but I want to show another mechanism. I want to combine static resources with dynamic resources. This is not possible with the WebServer. There is no integration between the WebServer and the RouteMatcher. Therefore I have written a few functions. We start we the functions that handle the request for static resources. All resources are located in the static folder. The following two functions return the index.html and content requested starting with /static. These are actual files on the file system and also treated that way using the sendFile function.

routeMatcher.get("/") { req ->
    req.response.sendFile("traderclient/static/index.html")
}
routeMatcher.getWithRegEx("^\\/static\\/.*") { req ->
    req.response.sendFile("traderclient/" + req.path.substring(1))
}

Notice the way we use getWithRegEx to map all static content paths to the requested files that start with /static/.

The next step is more interesting. How to we render a webpage with the obtained executed trades. The next code block shows the new function.

routeMatcher.get("/trades") {req ->
    def query = ["action":"find","collection":"tradeExecutedEntry","matcher":[:]]
    eventBus.send("vertx.mongopersistor",query) {message ->
        def buffer = new Buffer();
        buffer.appendBytes(new File('traderclient/header.tpl').readBytes())

        buffer.appendString("<table class='table table-striped'><thead><tr><th>Company</th><th>amount</th><th>prize</th></tr></thead><tbody>")
        def results = message.body.results
        results.each { item ->
            buffer.appendString("<tr><td>${item.companyName}</td><td>${item.tradeCount}</td><td>${item.tradePrice}</td></tr>")
        }
        buffer.appendString("</tbody></table>")

        buffer.appendBytes(new File('traderclient/footer.tpl').readBytes())
        req.response.end buffer
    }
}

First we create the buffer object and store the header.tpl file in it followed by the table with the items and the footer.tpl. As you can see this is a lot of text for creating a nice looking page.

A big disadvantage is that the server now generates the html, the browser is a lot better in doing this. Therefore I want to take the next step and use the WebSocket technology to render pages.

Use WebSockets to send data to the browser

Since WebSockets are not yet supported by all browsers, vert.x has support for the socks library. The framework facilitates creating a socks server that is connected to the HttpServer. We start we a clean groovy script. The following code block shows the initialization of the HttpServer together with a RouteMatcher that takes care of the static content.

def server = vertx.createHttpServer()
def routeMatcher = new RouteMatcher()

routeMatcher.get("/") { req ->
    req.response.sendFile("traderclient/static/socketDemo.html")
}

routeMatcher.getWithRegEx("^\\/static\\/.*") { req ->
    req.response.sendFile("traderclient/" + req.path.substring(1))
}

server.requestHandler(routeMatcher.asClosure())

This code has been discussed before. The next part is initializing the Sockjs server. This can be done very easily.

vertx.createSockJSServer(server).bridge(prefix: '/eventbus', [[:]])
server.listen(9090)

The next part on the server is to periodically check the mongo datastore for trades and send these trades to registered listeners.

EventBus eb = vertx.eventBus
vertx.setPeriodic(1000l) {
    def query = ["action": "find", "collection": "tradeExecutedEntry", "matcher": [:]]

    eb.send("vertx.mongopersistor", query) {message ->
        eb.send("updates.trades", message.body)
    }
}

As you can see, vert.x comes with a function to periodically do something. With these lines we call the mongo persisters via the event bus. The callback calls the even but again and send the data to the address updates.trades. This address is important for the client that needs to register a subscriber.

Now we can move on to the client. I am not going to show the complete page. Refer to the sources later on. The next code block shows the javascript code required to open the connection using sockjs and to subscribe to an address. The subscriber also defines the callback to use when a message is received.

    var eb = null;

    function subscribe() {
        if (eb) {
            eb.registerHandler("updates.trades", function (msg, replyTo) {
                var results = msg.results;
                $('#tradesBody').text('');
                for (var i = 0; i < results.length; i++) {
                    var item = results[i];
                    $('#tradesBody').append("<tr><td>" + item.companyName + "</td><td>" + item.tradeCount + "</td><td>" + item.tradePrice + "</td></tr>");
                }
                $('#lastUpdate').text(" " + new Date());
            });
        }
    }

    function closeConn() {
        if (eb) {
            eb.close();
        }
    }

    function openConn() {
        if (!eb) {
            eb = new vertx.EventBus("http://localhost:9090/eventbus");

            eb.onopen = function () {
                $("#connectionStatus").text("Connected");
                subscribe();
            };

            eb.onclose = function () {
                $("#connectionStatus").text("Not connected");
                eb = null;
            };
        }
    }

    $(document).ready(function () {
        $("#closeButton").click(function () {
            closeConn();
        });

        $("#connectButton").click(function () {
            openConn();
        });
    });

The openConn function creates a connection to the event bus at a certain url. We use jQuery to change the status to connected and Not connected. We also call the subscribe function to subscribe to the trades.updates address. The subscribe function registers a handler for receiving a message on the mentioned address. The callback receives the results. Within the callback we loop over the results and using jQuery we replace the current contents of the table body.

Now we have a webpage that is being updated by the server when it is connected. The timer can be used to see things are changing. I use the mongodb collections from the Axon-Trader sample. In the nearby future I want to create an application with vert.x that responds to events coming from the axon event bus. But more on this in a future blog post.

Problems during development

Caused by: groovy.lang.MissingMethodException: No signature of method: org.vertx.java.core.http.RouteMatcher.noMatch() is applicable for argument types: (App$_run_closure3) values: [App$_run_closure3@77587f0]
Possible solutions: noMatch(org.vertx.java.core.Handler), patch(java.lang.String, org.vertx.java.core.Handler), each(groovy.lang.Closure)

Seems to be a problem that is already resolved in the master, but not in the current release. This error is found when trying to use the noMatch method of the RouteMatcher.

mei 31, 2012 10:00:48 AM org.vertx.java.core.logging.impl.JULLogDelegate error
SEVERE: matcher must be specified
[message:matcher must be specified, status:error]

The problem was an error in the query. You need to provide a matcher, but how to do it? The following line show the query with at the end the matcher that you must configure, even if you want to match everything.

def query = ["action":"find","collection":"tradeExecutedEntry","matcher":[:]]

Concluding

I really like what vert.x is doing. With very little code you can create a very responsive application. I really like the WebSocket and sockjs integration. The amount of modules are limited, but more will come in the nearby future. Check the references for more information about vert.x

References

• Manual(s) for vert.x
• AxonTrader sample
• Sources of the sample




The post First steps with vert.x: creating a WebSocket sample appeared first on Gridshore.

I started thinking about putting the sources in a version control system. Since I like Git and have a Github account I wanted to store the files in Git. But I had some other requirements.

• I do not want to manually copy the changed files to the server.
• I am limited to ftp access, the hosted does not provide ssh access.
• I want to be able to make quick changes from multiple laptops.
• I want to have the changes taken from Git but also changes that have not yet been committed if I wanted to.
• I have no requirements for removing files, this does not happen a lot.




Since I am on a Mac and have some very basic experience with Bash, I decided to use Bash. I did have some issues with FTP, but more on that later on.

If you are curious about the websites I have created:

Yvonne van der Mey ~ Wildlife Photographer

Nico Bulder ~ Wildlife Painter

Repository layout

I want to support multiple sites in one repository. I give each site it’s own folder. The root of the project contains the deploy.lib file. This file contains the reusable code, which is almost everything. Each website folder contains a deploy.sh file. In the website folder we also have a file ftpconfig. This file is discussed later on when discussing the NcFtp client.

I do not store the complete wordpress installation in git. I most of store the template that you can find under wp-content/themes/yvonne. But I also add files that I added to the wordpress install like the favicon.ico. The idea is that I copy all changes over the existing wordpress installation. Like I stated before, I am not going to remove files. Using this mechanism I can also add custom plugins in the same way.

Explaining Git commands

The sources are stored in Git. This makes it easier to track changes and it helps me doing stuff the right way. So what is the right way? I want to deploy files that have changes in commits. Therefore I store the sha1 key of the last commit that I have deployed in a file. The next time I deploy changes, I read this file and deploy all changes after this commit. Sometimes however, you are debugging things and you do not want to continuously do commits. In that scenario you want to be able to deploy all changes in your local copy. This is also possible with the script.

Changes based on commits

Git has good support for finding more information on commits. There are multiple commands that can do the trick. For the most clean output came from the following command:

git show --pretty="format:" --name-only $startingfrom..HEAD

The $startingfrom is the sha1 key obtained through a provided parameter or from the file that was written the last time we did a deploy.

SHA1 of the last commit

We want to keep track of the changes that have already been deployed to the server. Since we are basing our deploys git commits, we want to obtain the sha1 key of the last commit. Git provides are daily easy way to obtain this key.

git log -1 --pretty=format:%H > $shaversion

With these commands we know enough to obtain changes after the last deploy. But what if we are trying something and want to make it possible to deploy files that have not yet been committed.

Changes in the local workspace

Changes that are not yet committed are called changes in the workspace of git. Again provides a good way to obtain the names of the files than have changes in the local workspace. You can check them against he HEAD.

git diff HEAD --name-only

That is it. Now we have the power to create a list of files that have changed. How can we copy files to the server using ftp.

NcFtp

The bash script creates a folder with a nested folder structure. The structure contains files on multiple levels. We want to use one command to recursively ftp all files to the server. Well, that is a hassle. First you have to overcome issue with ftp and logging in. Using the .netrc file this is easy to overcome on the mac. Still this does not resolve all our issues. Ftp does not support recursive ftp in a folder. You can use mput, but this only works for files in a folder. Not in a nested folder.

There are other options for ftp. I decided to use ncftp client software. Easy to install using Homebrew or Macports. I store the ftp connection data in a file that I configure in the ncftpput command. THe following command recursively copies the sources in the provided local folder to the provided remote folder.

The ftpconfig file must contain three properties: host, user, pass. The following command shows how you can copy the content of a directory to the server.

ncftpput -f ftpconfig -R mainwebsite_html deploy/yvonne/*

Doing bash

Time to put the pieces together. The glue is provided by bash.

Site specific configuration

Most of the bash code is shared between the different websites. That code is in the deploy.lib file. Of course you can place all code in one file. I have just one parameter and the included of the library file in the specific site deploy.sh file.

sitename="yvonne"
. ../deploy.lib
exit 1

Handling input

You can provide a few options to the script to influence what it does. It is good practice to show the options in a usage or help method. The following code block shows this method as well as the handling of the provided arguments.

function usage() {
	echo "Usage: `basename $0` [--nocommit] {GIT REVISION SHA}"
	echo "    optional flags: "
	echo "        --help, --usage # show this file"
	echo "        --nocommit #By default only commits are used, with this flag we use changes in your workspace."
	echo "        --sha1 <value> #Uses the commits from the provided value to the HEAD. Default reads the sha1.txt file."
}

commitonly=true
while true; do
	# case $# in 0) usage; exit 2 ;; esac
	case $1 in
		--help)				usage; exit 1 ;; esac
	case $1 in
		--usage)			usage; exit 1 ;; esac
	case $1 in
		--nocommit)			shift; commitonly=false ;; esac
	case $1 in
		--sha1)				shift; startingfrom=$1; shift ;;
		*) break ;;
	esac
done

The default is to gather all changes since the last commit that got deployed using the sha1 key in the sha1.tx file. By providing the –nocommit flag you can tell the script to switch to gathering changes in the local workspace. By providing the argument –sha1, you can pass another sha1 key to obtain changes for.

Obtaining the right changes

To find the files that have changed, we create a file containing the path to a file that is changed per line. By looping through these lines we can copy all files that we want to deploy. The next code block creates this file, but it has two side effects. It also creates a version.php that contains the date of the deploy as well as all the files that have been deployed. The second site effect is creating the sha1.txt file that contains the sha1 key of the last commit that has been deployed.

filelist="files-$(date +"%s").txt"
mkdir -p "deploy/$sitename"
versionphp="deploy/$sitename/version.php"
shaversion="sha1.txt"

if $commitonly ; then
	if [[ ${#startingfrom} -lt 1 ]]; then
		echo "Need to reed the sha1.txt file"
		while read line; do
			startingfrom=$(echo $line);
		done < $shaversion
	fi
	echo "Using commits for changes: $startingfrom"

	git show --pretty="format:" --name-only $startingfrom..HEAD > $filelist
	echo "Committed changes: $(date)" > $versionphp
	echo "<br>Last commit: $startingfrom" >> $versionphp
	git log -1 --pretty=format:%H > $shaversion
	echo "" >> $shaversion
else
	echo "We only use changes in the workspace"
	git diff HEAD --name-only > $filelist
	echo "Local changes: $(date)" > $versionphp
fi

Line 6 determines if the starting from parameter has been provided by an argument to the script. We determine if the length of the value is smaller than 1. If so we obtain it from the file sha1.txt. I think this was the hardest trick.

Ftp the changed files

Now we have a file that contains all the names of the files that have changed. The next block loops over the files and filters a few of them based on the website we are working for at the moment. If we have files left at the end we use the ncftpput command to actually ftp them.

ftpfiles=false
for file in $(<$filelist); do
	if [[ $file == $sitename* && $file != $sitename/deploy* && $file != $sitename/sha1.txt ]]; then
		echo "$file"
		echo "<br>$file" >> $versionphp
		mkdir -p "deploy/$(dirname $file)"
		cp "../$file" "deploy/$file"
		ftpfiles=true
	fi
done

if $ftpfiles ; then
	ncftpput -f ftpconfig -R mainwebsite_html deploy/yvonne/*
else
	echo "Nothing to ftp."
fi

Concluding

I am now able to adjust my workflow. I can make the changed locally, commit them into git before I actually automatically deploy them to the server. It is nice to see what you can do with a little bit of bash. If you have improvements please let me know.

The post Deployments with Git and Bash on a Mac appeared first on Gridshore.

Incubator Status

The mongo event store is becoming fully functional right now. At the moment the event store is functional for the 1.3.x branch of axon. We have a working example “The Axon Trader” using the event store. We do not have it running in a project in production yet. If you are considering to do use it in production please let us know about your findings.

The planning is to drop the incubator status starting Axon 2.0. Using the mentioned sample we are going to do some stress testing and we are thinking about improvements. More on these in the Next steps section.

Connecting to Mongo

MongoFactory

Mongo can be used as a single instance, which is not really suitable for a production environment. A better setup is to use a replication set of mongo instances. Using the MongoFactory this can be hidden for the other parts of the application.

The factory takes a list of server addresses, you can override the MongoOptions as well as the WriteConcern. Of course we take good defaults. There is an option to set the mode to single instance or multiple instance using a system property: axon.mongo.singleinstance. This system property takes precedence over other settings.

In the end, the MongoFactory is used to create a Mongo instance. This instance should be provided to the implementation of the MongoCollections class. More on this later on.

MongoOptionsFactory

The configuration of Mongo using the java driver is executed using the MongoOptions class. This factory is used to create such an object using defaults but with the possibility to provider other values for the most important properties. Some of the options you can configure are: number of connections per host, the connection time-out and auto-connect retry.

The collections in Mongo

Mongo uses collections to store data. The names of these collections that Axon uses for the event store as well as the saga store can be configured. Of course logical default values are provided as well. If you want to use other collections you can provide your own implementation of this interface or provide other names to the default implementation of the interface.

Overview of the classes

The following image gives you an overview of the important classes.

Event repository

The event repository, MongoEventStore, is responsible for storing and retrieving the events. Just like the other implementations of the event stores, you can provide your own event serializer. By default XStream is used to serialize the events.

The java driver for mongo makes it easy to store new documents in Mongo and query the existing documents. By using the EventStoreCollections object we obtain a reference to the actual collection and by calling insert we add new documents to the collection. To get an idea about the code, have a look at the following function of the event store.

public void appendEvents(String type, DomainEventStream events) {
    List<DBObject> entries = new ArrayList<DBObject>();
    while (events.hasNext()) {
        DomainEvent event = events.next();
        EventEntry entry = new EventEntry(type, event, eventSerializer);
        entries.add(entry.asDBObject());
    }
    eventStoreCollections.domainEventCollection().insert(entries.toArray(new DBObject[entries.size()]));
}

A similar method is available to store snapshot events.

It becomes interesting when reading events to recreate an aggregate. Before reading events that need to be applied, we first need to find the latest snapshot. The following code block gives an idea about finding the latest snapshot event.

private EventEntry loadLastSnapshotEvent(String type, AggregateIdentifier identifier) {
    DBObject mongoEntry = BasicDBObjectBuilder.start()
            .add(EventEntry.AGGREGATE_IDENTIFIER_PROPERTY, identifier.asString())
            .add(EventEntry.AGGREGATE_TYPE_PROPERTY, type)
            .get();
    DBCursor dbCursor = eventStoreCollections.snapshotEventCollection()
            .find(mongoEntry)
            .sort(new BasicDBObject(EventEntry.SEQUENCE_NUMBER_PROPERTY, -1))
            .limit(1);

    if (!dbCursor.hasNext()) {
        return null;
    }
    DBObject first = dbCursor.next();

    return new EventEntry(first);
}

Now you have seen the main features of the event store in Mongo. The next step is to store and load Saga instances.

Saga repository

The mongo part for Saga’s is even easier than the events. We do not have to account for snapshots, which makes it easier. Just use the identifier of a saga instance to store, load and delete a document. The repository supports storing the associations between Saga’s as required by Axon. It also supports the ResourceInjector if your saga needs other resources to do it’s job. Think about the command bus to send new commands.

The structure is the same as for the event store implementation. The SagaStoreCollections provides access to the collections in the mongo database as well as the database itself.

Next steps

We are constantly looking for improvements in the mongo implementation. Allard is working on a specific Mongo serializer. That way we are nog serializing to XML and storing that xml. We will store the content in a DBObject and that way in a basic mongo document structure. More on this will follow later on in another blog post.

As mentioned we are planning on doing some performance tests and experiment with indexes to improve performance.

I you have improvements, bugs or ideas feel free to file an issue on our Github project.

The post A MongoDB based Axon framework event store appeared first on Gridshore.

http://blog.dutchworks.nl/2011/12/13/embedding-rss-in-hippo-using-the-pipelines-feature/

The post Creating an rss feed using the latest features of Hippo appeared first on Gridshore.