i’m a lumberjaph

Lately I’ve been toying a lot with Plack and two Perl web framework: Tatsumaki and Dancer. I use both of them for different purposes, as their features complete each other.

Plack

If you don’t already know what Plack is, you would want to take a look at the following Plack resources:

• Plack (redesigned) website
• Plack documentation
• miyagawa’s screencast
• Plack advent calendar

As sukria is planning to talk about Dancer during the FPW 2010, I will probably do a talk about Plack.

After reading some code, I’ve started to write two middleware: the first one add ETag header to the HTTP response, and the second one provides a way to limit access to your application.

Plack::Middleware::ETag

This middleware is really simple: for each request, an ETag header is added to the response. The ETag value is a sha1 of the response’s content. In case the content is a file, it works like apache, using various information from the file: inode, modified time and size. This middleware can be used with Plack::Middleware::ConditionalGET, so the client will have the ETag information for the page, and when he will do a request next time, it will send an “if-modified” header. If the ETag is the same, a 304 response will be send, meaning the content have not been modified. This module is available on CPAN.

Let’s see how it works. First, we create a really simple application (we call it app.psgi):

#!/usr/bin/env perl
use strict;
use warnings;
use Plack::Builder;
 
builder {
    enable "Plack::Middleware::ConditionalGET";
    enable "Plack::Middleware::ETag";
    sub {
        [ '200', [ 'Content-Type' => 'text/html' ], ['Hello world'] ];
    };
};

Now we can test it:

> plackup app.psgi&
 
> curl -D - http://localhost:5000
HTTP/1.0 200 OK
Date: Sat, 03 Apr 2010 09:31:43 GMT
Server: HTTP::Server::PSGI
Content-Type: text/html
ETag: 7b502c3a1f48c8609ae212cdfb639dee39673f5e
Content-Length: 11
 
> curl -H "If-None-Match: 7b502c3a1f48c8609ae212cdfb639dee39673f5e" -D - http://localhost:5000
HTTP/1.0 304 Not Modified
Date: Sat, 03 Apr 2010 09:31:45 GMT
Server: HTTP::Server::PSGI
ETag: 7b502c3a1f48c8609ae212cdfb639dee39673f5e

Plack::Middleware::Throttle

With this middleware, you can control how many times you want to provide an access to your application. This module is not yet on CPAN, has I want to add some features, but you can get the code on github. There is four methods to control access:

• Plack::Middleware::Throttle::Hourly: how many times in one hour someone can access the application
• P::M::T::Daily: the same, but for a day
• P::M::T::Interval: which interval the client must wait between two query
• by combining the three previous methods

To store sessions informations, you can use any cache backend that provides get, set and incr methods. By default, if no backend is provided, it will store informations in a hash. You can easily modify the defaults throttling strategies by subclassing all the classes.

Let’s write another application to test it:

#!/usr/bin/env perl
use strict;
use warnings;
use Plack::Builder;
 
builder {
    enable "Plack::Middleware::Throttle::Hourly", max => 2;
    sub {
        [ '200', [ 'Content-Type' => 'text/html' ], ['Hello world'] ];
    };
};

$ curl -D - http://localhost:5000/
HTTP/1.0 200 OK
Date: Sat, 03 Apr 2010 09:57:40 GMT
Server: HTTP::Server::PSGI
Content-Type: text/html
X-RateLimit-Limit: 2
X-RateLimit-Remaining: 1
X-RateLimit-Reset: 140
Content-Length: 11
 
Hello world
 
$ curl -D - http://localhost:5000/
HTTP/1.0 200 OK
Date: Sat, 03 Apr 2010 09:57:40 GMT
Server: HTTP::Server::PSGI
Content-Type: text/html
X-RateLimit-Limit: 2
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 140
Content-Length: 11
 
Hello world
 
$ curl -D - http://localhost:5000/
HTTP/1.0 503 Service Unavailable
Date: Sat, 03 Apr 2010 09:57:41 GMT
Server: HTTP::Server::PSGI
Content-Type: text/plain
X-RateLimit-Reset: 139
Content-Length: 15
 
Over rate limit

Some HTTP headers are added to the response :

• X-RateLimit-Limit: how many request can be done
• X-RateLimit-Remaining: how many requests are available
• X-RateLimit-Reset: when will the counter be reseted (in seconds)

This middleware could be a very good companion to the Dancer REST stuff added recently.

another Tatsumaki application with Plack middlewares

To demonstrate the use of this two middleware, I’ve wrote a small application with Tatsumaki. This application fetch a page, parse it to find all the feeds declared, and return a JSON with the result.

GET http://feeddiscover.tirnan0g.org/?url=http://lumberjaph.net/blog/

will return

[{"href":"http://lumberjaph.net/blog/index.php/feed/","type":"application/rss+xml","title":"iâm a lumberjaph RSS Feed"}]

This application is composed of one handler, that handle only GET request. The request will fetch the url given in the url parameter, scrap the content to find the links to feeds, and cache the result with Redis. The response is a JSON string with the informations.

The interesting part is the app.psgi file:

 
my $app = Tatsumaki::Application->new( [ '/' => 'FeedDiscovery::Handler' ], );
 
builder {
    enable "Plack::Middleware::ConditionalGET";
    enable "Plack::Middleware::ETag";
    enable "Plack::Middleware::Throttle::Hourly",
        backend => Redis->new(
        server => '127.0.0.1:6379',
        ),
        max => 100;
    $app;
};

The application itself is really simple: for a given url, the Tatsumaki::HTTPClient fetch an url, I use Web::Scraper to find the link rel=”alternate” from the page, if something is found, it’s stored in Redis, then a JSON string is returned to the client.

This week, with Alexis’s help, I’ve been working on adding auto-(de)serialization to Dancer’s request. This features will be available in the next Dancer version, the 1.170 (which will be out before April).

The basic idea was to provides to developer a simple way to access data that have been send in a serialized format, and to properly serialize the response.

At the moment, the supported serializers are :

• Dancer::Serialize::JSON
• Dancer::Serialize::YAML
• Dancer::Serialize::XML
• Dancer::Serialize::Mutable

Configuring an application to use the serializer

To activate serialization in your application:

set serializer => 'JSON';

or in your configuration file:

serializer: "JSON"

A simple handler

Let’s create a new dancer application (you can fetch the source at github) :

dancer -a dancerREST
cd dancerREST
vim dancerREST.pm

package dancerREST;
use Dancer ':syntax';
 
my %users = ():
 
# create a new user
post '/api/user/' => sub {
    my $params = request->params;
    if ( $params->{name} && $params->{id} ) {
        if ( exists $users{ $params->{id} } ) {
            return { error => "user already exists" };
        }
        $users{$params->{id}} = {name => $params->{name}};
        return { id => $params->{id}, name => $params->{name} };
    }
    else {
        return { error => "name is missing" };
    }
};
 
true;

We can test if everything works as expected:

plackup app.psgi&
> curl -H "Content-Type: application/json" -X POST http://localhost:5000/api/user/ -d '{"name":"foo","id":1}'
# => {"name":"foo","id":"1"}

Now we add a method to fetch a list of users, and a method to get a specific user:

# return a specific user
get '/api/user/:id' => sub {
    my $params = request->params;
    if ( exists $users{ $params->{id} } ) {
        return $users{$params->{id}};
    }
    else {
        return { error => "unknown user" };
    }
};
 
# return a list of users
get '/api/user/' => sub {
    my @users;
    push @users, { name => $users{$_}->{name}, id => $_ } foreach keys %users;
    return \@users;
};

If we want to fetch the full list:

> curl -H "Content-Type: application/json" http://localhost:5000/api/user/
# => [{"name":"foo","id":"1"}]

and a specific user:

> curl -H "Content-Type: application/json" http://localhost:5000/api/user/1
# => {"name":"foo"}

The mutable serializer

The mutable serializer will try to load an appropriate serializer guessing from the Content-Type and Accept-Type header. You can also overload this by adding a content_type=application/json parameter to your request.

While setting your serializer to mutable, your let your user decide which format they prefer between YAML, JSON and XML.

And the bonus

Dancer provides now a new method to the request object : is_ajax. Now you can write something like

get '/user/:id' => sub {
    my $params = request->params;
    my $user    = $users{ $params->{id} };
    my $result;
    if ( !$user ) {
        _render_user( { error => "unknown user" } );
    }
    else {
        _render_user($user);
    }
};
 
sub _render_user {
    my $result  = shift;
    if ( request->is_ajax ) {
        return $result;
    }
    else {
        template 'user.tt', $result;
    }
}

If we want to simulate an AJAX query:

curl -H "X-Requested-With: XMLHttpRequest" http://localhost:5000/user/1

and we will obtain our result in JSON. But we can also test without the X-Requested-With:

curl http://localhost:5000/user/1

and the template will be rendered.

Hope you like this new features. I’ve also been working on something similar for Tatsumaki.

You may want to see the final version here: github explorer

For the last weeks, I’ve been working on the successor of CPAN Explorer. This time, I’ve decided to create some visualizations (probably 8) of the various communities using Github. I’m happy with the result, and will soon start to publish the maps (statics and interactives) with some analyses. I’m publishing two previews: the Perl community and the european developers. These are not final results. The colors, fonts, and layout may change. But the structure of the graphs will be the same. All the data was collected using the github API.

Each node on the graph represents a developer. When a developer “follows” another developer on github, a link between them is created. The color on the Perl community map represent the countries of the developer. One of the most visible things on this graph is that the japanese community is tighly connected and shares very little contact with the foreign developers. miyagawa obviously acts as a glue between japanese and worldwide Perl hackers.

The second graph is a little bit more complex. It represents the European developers on github. Here the colors represent the languages used by the developers. It appears that ruby is by far the most represented language on github, as it dominates the whole map. Perl is the blue cluster at the bottom of the map, and the green snake is… Python.

Thanks to bl0b for his suggestions :)

If you are using the version of SD hosted on github, you can now clone and pull issues very easily. First,

$ git config --global github.user franckcuny
$ git config --global github.token myapitoken

This will set in your .gitconfig your github username and api token. Now, when you clone or pull some issues using sd:

$ git sd clone --from github:sukria/Dancer

sd will check your .gitconfig to find your credentials.

Alexis (sukria) released Dancer 1.130 this weekend. Dancer is a small and nice web framework based on ruby’s sinatra.

Dancer have few dependancies (and it doesn’t depends anymore on CGI.pm). The path dispatching is done using rules declared with HTTP methods (get/post/put/delete), and they are mapped to a sub-routine which is returned as the response to the request. Sessions are supported, and two template engines (one of them is Template Toolkit) comes with the Core. Dancer::Template::MicroTemplate is also available on CPAN if you need a light template engine.

You can easily test it with a simple script

#!/usr/bin/env perl
use Dancer;
 
get '/' => sub {
  return "dancer";
}
 
get '/:name' => sub {
  return params->{name} ." is dancing";
}
 
dance;

and execute this script, point your browser to http://127.0.0.1:3000, and voila.

Dancer provides also a small helper to write a new application:

dancer -a MyApplication

If you create an application with this script, an app.psgi file will be created. You can now execute plackup –port 8080 (which comes with Plack the Perl Web Server) and test if everything works fine:

curl http://localhost:8080

This release remove some components from the core and they are now available as differents CPAN distributions. Two new keyword have also been added, header and prefix.

If you want to read more about Dancer:

• Dancer’s documentation
• rewiew by xsawyerx
• gugod’s review
• sukria’s blog