github explorer - a preview

March 7th, 2010

For the last weeks, I’ve been working on the successor of CPAN Explorer. This time, I’ve decided to create some visualizations (probably 8) of the various communities using Github. I’m happy with the result, and will soon start to publish the maps (statics and interactives) with some analyses. I’m publishing two previews: the Perl community and the european developers. These are not final results. The colors, fonts, and layout may change. But the structure of the graphs will be the same. All the data was collected using the github API.

the Perl community on github

Each node on the graph represents a developer. When a developer “follows” another developer on github, a link between them is created. The color on the Perl community map represent the countries of the developer. One of the most visible things on this graph is that the japanese community is tighly connected and shares very little contact with the foreign developers. miyagawa obviously acts as a glue between japanese and worldwide Perl hackers.

European developers on github

The second graph is a little bit more complex. It represents the European developers on github. Here the colors represent the languages used by the developers. It appears that ruby is by far the most represented language on github, as it dominates the whole map. Perl is the blue cluster at the bottom of the map, and the green snake is… Python.

Thanks to bl0b for his suggestions :)

SD and github

February 3rd, 2010

If you are using the version of SD hosted on github, you can now clone and pull issues very easily. First,

$ git config --global github.user franckcuny
$ git config --global github.token myapitoken

This will set in your .gitconfig your github username and api token. Now, when you clone or pull some issues using sd:

$ git sd clone --from github:sukria/Dancer

sd will check your .gitconfig to find your credentials.

Dancer 1.130

January 31st, 2010

Alexis (sukria) released Dancer 1.130 this weekend. Dancer is a small and nice web framework based on ruby’s sinatra.

Dancer have few dependancies (and it doesn’t depends anymore on CGI.pm). The path dispatching is done using rules declared with HTTP methods (get/post/put/delete), and they are mapped to a sub-routine which is returned as the response to the request. Sessions are supported, and two template engines (one of them is Template Toolkit) comes with the Core. Dancer::Template::MicroTemplate is also available on CPAN if you need a light template engine.

You can easily test it with a simple script

#!/usr/bin/env perl
use Dancer;
 
get '/' => sub {
  return "dancer";
}
 
get '/:name' => sub {
  return params->{name} ." is dancing";
}
 
dance;

and execute this script, point your browser to http://127.0.0.1:3000, and voila.

Dancer provides also a small helper to write a new application:

dancer -a MyApplication

If you create an application with this script, an app.psgi file will be created. You can now execute plackup –port 8080 (which comes with Plack the Perl Web Server) and test if everything works fine:

curl http://localhost:8080

This release remove some components from the core and they are now available as differents CPAN distributions. Two new keyword have also been added, header and prefix.

If you want to read more about Dancer:

Tatsumaki, or how to write a nice webapp in less than two hours

December 21st, 2009

Until today, I had a script named “lifestream.pl”. This script was triggered via cron once every hour, to fetch various feeds from services I use (like github, identi.ca, …) and to process the result through a template and dump the result in a HTML file.

Today I was reading Tatsumaki’s code and some examples (Social and Subfeedr). Tatsumaki is a “port” tornado (a non blocking server in Python), based on Plack and AnyEvent. I though that using this to replace my old lifestream script would be a good way to test it. Two hours later I have a complete webapp that works (and the code is available here).

The code is really simple: first, I define an handler for my HTTP request. As I have only one things to do (display entries), the handler is really simple:

package Lifestream::Handler;   
use Moose;                     
extends 'Tatsumaki::Handler';  
 
sub get {                      
    my $self = shift;          
    my %params = %{$self->request->params};
    $self->render( 'lifestream.html', {
        memes    => $self->application->memes($params{page}),
        services => $self->application->services
    });
}
1;

For all the get request, 2 methods are called : memes and services. The memes get a list of memes to display on the page. The services get the list of the various services I use (to display them on a sidebar).

Now, as I don’t want to have anymore my lifestream.pl script in cron, I will let Tatsumaki do the polling. For this, I add a service to my app, which is just a worker.

package Lifestream::Worker;    
use Moose;                     
extends 'Tatsumaki::Service';  
use Tatsumaki::HTTPClient;     
...
sub start {
    my $self = shift;
    my $t; $t = AE::timer 0, 1800, sub {
        scalar $t;
        $self->fetch_feeds;
    };
}
....
sub fetch_feeds {
    my ($self, $url) = @_;
    Tatsumaki::HTTPClient->new->get( $url, sub { #do the fetch and parsing stuff });
}

From now, every 60 minutes, feeds will be checked. Tatsumaki::HTTPClient is a HTTP client based on AnyEvent::HTTP.

Let’s write the app now

package Lifestream;            
 
use Moose;
extends "Tatsumaki::Application";
 
use Lifestream::Handler;       
use Lifestream::Worker;        
...
sub app {
    my ( $class, %args ) = @_;
    my $self = $class->new( [ '/' => 'Lifestream::Handler', ] );
    $self->config( $args{config} ); 
    $self->add_service( Lifestream::Worker->new( config => $self->config ) );
    $self;
}
...
sub memes {
...
}
 
sub services {
....
}

The memes and services method called from the handler are defined here. In the app method, I “attch” the “/” path to the handler, and I add the service.

and to launch the app

my $app = Lifestream->app( config => LoadFile($config) );
require Tatsumaki::Server;      
Tatsumaki::Server->new(
    port => 9999,
    host => 0,
)->run($app);

And that’s it, I now have a nice webapp, with something like only 200 LOC. I will keep playing with Tatsumaki as I have more ideas (and probably subfeedr too). Thanks to miyagawa for all this code.

MooseX::Net::API

December 20th, 2009

Net::Twitter

I’ve been asked for $work to write an API client for backtype, as we plan to integrate it in one of our services. A couple of days before I was reading the Net::Twitter source code, and I’ve found interesting how semifor wrote it.

Basically, what Net::Twitter does is this: for each API method, there is a twitter_api_method method, where the only code for this method is an API specification of the method. Let’s look at the public timeline method:

twitter_api_method home_timeline => (
    description => <<'',
Returns the 20 most recent statuses, including retweets, posted by the
authenticating user and that user's friends. This is the equivalent of
/timeline/home on the Web.
 
    path      => 'statuses/home_timeline',
    method    => 'GET',
    params    => [qw/since_id max_id count page/],
    required  => [],
    returns   => 'ArrayRef[Status]',
);

The twitter_api_method method is exported with Moose::Exporter. It generates a sub called home_timeline that is added to the class.

MooseX::Net::API

As I’ve found this approch nice and simple, I thought about writing a little framework to easily write API client this way. I will show how I’ve write a client for the Backtype API using this (I’ve wrote some other client for private API at works too).

Backtype API

First we defined our class:

package Net::Backtweet;        
 
use Moose;
use MooseX::Net::API;

MooseX::Net::API export two methods: net_api_declare and net_api_method. The first method is for all the paramters that are common for each method. For Backtype, I’ll get this:

net_api_declare backtweet => (
    base_url    => 'http://backtweets.com',
    format      => 'json',
    format_mode => 'append',
);

This set

  • the base URL for the API
  • the format is JSON
  • some API use an extension at the name of the method to determine the format. “append” do this.

Right now three formats are supported: xml json and yaml. Two modes are supported: append and content-type.

Now the net_api_method method.

net_api_method backtweet_search => (
    path     => '/search',
    method   => 'GET',
    params   => [qw/q since key/],  
    required => [qw/q key/],
    expected => [qw/200/],
);
  • path: path for the method (required)
  • method: how to acces this resource (GET POST PUT and DELETE are supported) (required)
  • params: list of parameters to access this resource (required)
  • required: which keys are required
  • expected: list of HTTP code accepted

To use it:

my $backtype = Net::Bactype->new();
my $res = $backtype->backtweet_search(q => "http://lumberjaph.net", key => "foo");
warn Dump $res->{tweets};

MooseX::Net::API implementation

Now, what is done by the framework. The net_api_declare method add various attributes to the class:

  • api_base_url: base URL of the API
  • api_format: format for the query
  • api_format_mode: how the format is used (append or content-type)
  • api_authentication: if the API requires authentication
  • api_username: the username for accessing the resource
  • api_password: the password
  • api_authentication: does the resource requires to be authenticated

It will also apply two roles, for serialization and deserialization, unless you provides your own roles for this. You can provides your own method for useragent and authentication too (the module only do basic authentication).

For the net_api_method method, you can overload the authentication (in case some resources requires authentication). You can also overload the default code generated.

In case there is an error, an MooseX::Net::API::Error will be throw.

Conclusion

Right now, this module is not finished. I’m looking for suggestions (what should be added, done better, how I can improve stuff, …). I’m not aiming to handle all possibles API, but at least most of the REST API avaible. I’ve uploaded a first version of MooseX::Net::API and Net::Backtype on CPAN, and the code is available on github.

For testing purpose, i’ve set a dumb REST service here (the code is here). I will update this service to add more tests to MX::Net::API.