i’m a lumberjaph

Net::Twitter

I’ve been asked for $work to write an API client for backtype, as we plan to integrate it in one of our services. A couple of days before I was reading the Net::Twitter source code, and I’ve found interesting how semifor wrote it.

Basically, what Net::Twitter does is this: for each API method, there is a twitter_api_method method, where the only code for this method is an API specification of the method. Let’s look at the public timeline method:

twitter_api_method home_timeline => (
    description => <<'',
Returns the 20 most recent statuses, including retweets, posted by the
authenticating user and that user's friends. This is the equivalent of
/timeline/home on the Web.
 
    path      => 'statuses/home_timeline',
    method    => 'GET',
    params    => [qw/since_id max_id count page/],
    required  => [],
    returns   => 'ArrayRef[Status]',
);

The twitter_api_method method is exported with Moose::Exporter. It generates a sub called home_timeline that is added to the class.

MooseX::Net::API

As I’ve found this approch nice and simple, I thought about writing a little framework to easily write API client this way. I will show how I’ve write a client for the Backtype API using this (I’ve wrote some other client for private API at works too).

Backtype API

First we defined our class:

package Net::Backtweet;        
 
use Moose;
use MooseX::Net::API;

MooseX::Net::API export two methods: net_api_declare and net_api_method. The first method is for all the paramters that are common for each method. For Backtype, I’ll get this:

net_api_declare backtweet => (
    base_url    => 'http://backtweets.com',
    format      => 'json',
    format_mode => 'append',
);

This set

• the base URL for the API
• the format is JSON
• some API use an extension at the name of the method to determine the format. “append” do this.

Right now three formats are supported: xml json and yaml. Two modes are supported: append and content-type.

Now the net_api_method method.

net_api_method backtweet_search => (
    path     => '/search',
    method   => 'GET',
    params   => [qw/q since key/],  
    required => [qw/q key/],
    expected => [qw/200/],
);

• path: path for the method (required)
• method: how to acces this resource (GET POST PUT and DELETE are supported) (required)
• params: list of parameters to access this resource (required)
• required: which keys are required
• expected: list of HTTP code accepted

To use it:

my $backtype = Net::Bactype->new();
my $res = $backtype->backtweet_search(q => "http://lumberjaph.net", key => "foo");
warn Dump $res->{tweets};

MooseX::Net::API implementation

Now, what is done by the framework. The net_api_declare method add various attributes to the class:

• api_base_url: base URL of the API
• api_format: format for the query
• api_format_mode: how the format is used (append or content-type)
• api_authentication: if the API requires authentication
• api_username: the username for accessing the resource
• api_password: the password
• api_authentication: does the resource requires to be authenticated

It will also apply two roles, for serialization and deserialization, unless you provides your own roles for this. You can provides your own method for useragent and authentication too (the module only do basic authentication).

For the net_api_method method, you can overload the authentication (in case some resources requires authentication). You can also overload the default code generated.

In case there is an error, an MooseX::Net::API::Error will be throw.

Conclusion

Right now, this module is not finished. I’m looking for suggestions (what should be added, done better, how I can improve stuff, …). I’m not aiming to handle all possibles API, but at least most of the REST API avaible. I’ve uploaded a first version of MooseX::Net::API and Net::Backtype on CPAN, and the code is available on github.

For testing purpose, i’ve set a dumb REST service here (the code is here). I will update this service to add more tests to MX::Net::API.

For $work, I’ve been working on a job queue system, using Moose, Catalyst (for a REST API) and DBIx::Class to store the jobs and some meta (yeah I know, there is not enough job queue system already, the world really needs a new one …).

Basicaly, I’ve got a XXX::Worker class that all the workers extends. This class provide methods for fetching job, add a new job, mark a job as fail, retry, …

The main loop in the XXX::Worker class look like this:

# $context is a hashref with some info the job or method may need
while(1) {
    my @jobs = $self->fetch_jobs();
    foreach my $job (@jobs) {
        my $method = $job->{funcname};
        $self->$method($context, $job);
    }
    $self->wait;
}

and the worker look like this

package MyWorker;
use Moose;
extends 'XXX::Worker';
 
sub foo {
    my ($self, $context, $job) = @_;
    # do something
    $self->job_success();
}

But as I’m using Moose, I want to add more sugar to the syntax, so writing a new worker would be really more easy.

Here comes Devel::Declare.

The syntax I want for my worker is this one:

work foo {
    $self->logger->info("start to work on job");
    # do something with $job
}
 
work bar {
    # do something with $job
}
 
success foo {
    $self->logger->info("woot job success");
}
 
fail bar {
    $self->logger->info("ho noez this one failed");
}

Where with ‘work‘ I write the code the writer will execute on a task, ‘success‘, a specific code that will be executed after a job is marked as successfull, and ‘fail‘ for when the job fail.

I will show how to add the ‘work‘ keyword. I start by writing a new package :

XXX::Meta:
 
package XXX::Meta;
 
use Moose;
use Moose::Exporter;
use Moose::Util::MetaRole;
 
use Devel::Declare;
 
use XXX::Meta::Class;
use XXX::Keyword::Work;
 
Moose::Exporter->setup_import_methods();
 
sub init_meta {
    my ( $me, %options ) = @_;
 
    my $for = $options{for_class};
 
    XXX::Keyword::Work->install_methodhandler( into => $for, );
 
    Moose::Util::MetaRole::apply_metaclass_roles(
        for_class       => $for,
        metaclass_roles => ['XXX::Meta::Class'],
    );
 
}
 
1;

The init_meta method is provided by Moose: (from the POD)

The init_meta method sets up the metaclass object for the class specified by for_class. This method injects a a meta accessor into the class so you can get at this object. It also sets the class’s superclass to base_class, with Moose::Object as the default.

So I inject into the class that will use XXX::Meta a new metaclass, XXX::Meta::Class.

Let’s take a look to XXX::Meta::Class:

package XXX::Meta::Class;
 
use Moose::Role;
use Moose::Meta::Class;
use MooseX::Types::Moose qw(Str ArrayRef ClassName Object);
 
has work_metaclass  => (
    is      => 'ro',
    isa     => Object,
    builder => '_build_metaclass',
    lazy    => 1,
);
 
has 'local_work' => (
    traits     => ['Array'],
    is         => 'ro',
    isa        => ArrayRef [Str],
    required   => 1,
    default    => sub { [] },
    auto_deref => 1,
    handles    => { '_add_work' => 'push', }
);
 
sub _build_metaclass {
    my $self = shift;
    return Moose::Meta::Class->create_anon_class(
        superclasses => [ $self->method_metaclass ],
        cache        => 1,
    );
}
 
sub add_local_method {
    my ( $self, $method, $name, $code ) = @_;
 
    my $method_name = $method . "_" . $name;
    my $body        = $self->work_metaclass->name->wrap(
        $code,
        original_body => $code,
        name          => $method_name,
        package_name  => $self->name,
    );
 
    my $method_add = "_add_" . $method;
    $self->add_method( $method_name, $body );
    $self->$method_add($method_name);
}
 
1;

Here I add to the ->meta provided by Moose ‘local_work‘, which is an array that contains all my ‘work‘ methods. So each time I do something like

work foo {
}
 
work bar {
}

in my worker, I add this method to ->meta->local_work.

And the class for our keyword work:

package XXX::Keyword::Work;
 
use strict;
use warnings;
 
use Devel::Declare ();
use Sub::Name;
 
use base 'Devel::Declare::Context::Simple';
 
sub install_methodhandler {
    my $class = shift;
    my %args  = @_;
    {
        no strict 'refs';
        *{ $args{into} . '::work' } = sub (&) { };
    }
 
    my $ctx = $class->new(%args);
    Devel::Declare->setup_for(
        $args{into},
        {
            work => {
                const => sub { $ctx->parser(@_) }
            },
        }
    );
}
 
sub parser {
    my $self = shift;
    $self->init(@_);
 
    $self->skip_declarator;
    my $name = $self->strip_name;
    $self->strip_proto;
    $self->strip_attrs;
 
    my $inject = $self->scope_injector_call();
    $self->inject_if_block(
        $inject . " my (\$self, \$content, \$job) = \@_; " );
 
    my $pack = Devel::Declare::get_curstash_name;
    Devel::Declare::shadow_sub(
        "${pack}::work",
        sub (&) {
            my $work_method = shift;
            $pack->meta->add_local_method( 'work', $name, $work_method );
        }
    );
    return;
}
 
1;

The install_methodhandler add the work keyword, with a block of code. This code is sent to the parser, that will add more sugar. With the inject_if_block, I inject the following line

my ($self, $context, $job) = @_;

as this will always be my 3 arguments for a work method.

Now, for each new worker, I write something like this:

package MyWorker;
use Moose;
extends 'XXX::Worker';
use XXX::Meta;
 
work foo {
}

The next step is too find the best way to reduce the first four lines to two.

(some of this code is ripped from other modules that use Devel::Declare. The best way to learn what you can do with this module is to read code from other modules that use it)

Today I presented a talk about Moose at OSDC.fr. The slides are available here.
And big thanks to my friend Morgan for his illustration of the batmoose :)

You can apply a role to a Moose object. You can do something like

#!/usr/bin/perl -w
use strict;
use feature ':5.10';
 
{
    package foo;
    use Moose::Role;
    sub baz { say 'i can haz baz'; }
 
    package bar;
    use Moose;
    1;
}
 
my $test = bar->new;
say "i can't haz baz" if !$test->can("baz");
foo->meta->apply($test);
$test->baz;

with the following output

i can't haz baz
i can haz baz

Or from the object

#!/usr/bin/perl -w
use strict;
use feature ':5.10';
 
{
    package foo;
    use Moose::Role;
    sub baz { say 'i can haz baz'; }
 
    package bar;
    use Moose;
    sub apply_a_role { my ($self, $role) = @_; $role->meta->apply($self); }
    1;
}
 
my $test = bar->new;
say "i can't haz baz" if !$test->can("baz");
$test2->apply_a_role('foo');
$test2->baz;

With the same results.

Yesterday, one of our interns asked me a question about private method in Moose. I told him that for Moose as for Perl, there is no such things as private method. By convention, methods prefixed with ‘_’ are considered private.

But I was curious to see if it would be something complicated to implement in Moose. First, I’ve started to look at how the ‘augment’ keyword is done. I’ve then hacked Moose directly to add the private keyword. After asking advice to nothingmuch, he recommended me that I implement this in a MooseX::* module instead. The result is here.

From the synopsis, MooseX::MethodPrivate do:

package Foo;
use MooseX::MethodPrivate;
 
private 'foo' => sub {
    ...
}
 
protected 'bar' => sub {
    ...
}
 
 ...
 
my $foo = Foo->new;
$foo->foo; # die, can't call foo because it's a private method
$foo->bar; # die, can't call bar because it's a protected method
 
package Bar;
use MooseX::MethodPrivate;
extends qw/Foo/;
 
sub baz {
    my $self = shift;
    $self->foo; # die, can't call foo because it's a private method
    $self->bar; # ok, can call this method because we extends Foo and
                     # it's a protected method
}

I was surprised to see how easy it’s to extend Moose syntax. All I’ve done was this:

Moose::Exporter->setup_import_methods( with_caller => [qw( private protected )], );

and write the ‘private’ and ‘protected’ sub. I’m sure there is some stuff I can do to improve this, but for a first test, I’m happy with the result and still amazed how easy it was to add this two keywords.