Distributed caching and message queues are basic building blocks of distributed computing. Memcached is clearly the best of breed for caching. For message queues however the choices are many and vary in features, reliability and ease of use.
I needed a simple and fast queue with only basic reliability (lost messages don’t cost much). Memcached offers some special operations that make it possible to build queues… all you need is a couple counters. Here’s the basic idea:
First create head and tail keys to track the unread messages in the queue. Then using the atomic incr() operation memcached provides, increment the head and use that number as the key for a new message in the queue. Increment the tail and use that the key of the message of the next message to read. You can calculate the current queue size by subtracting the head and tail counters.
On the downside you may get no messages when you call recv() if the cache is full. Also, there is also no way of knowing what happens to a message once you recv() it. On the upside you can drop in a service like memcachedb for persistent storage.
There are similar queues built using memcached like sparrow.
Here is the perl implementation, should be easy to port to any language memcached provides clients for…
package MemQueue;
use strict;
use warnings;
use Cache::Memcached;
use constant PREFIX => "MEMQUEUE_";
sub new
{
my $cn = shift;
my $name = shift;
die("queue name required") unless defined $name;
my @servers = @_;
my $self = {};
$self->{name} = PREFIX().$name;
$self->{head} = PREFIX().$name."_head";
$self->{tail} = PREFIX().$name."_tail";
$self->{memd} = new Cache::Memcached( servers => @servers );
#create queue
$self->{memd}->add( $self->{head}, 0 );
$self->{memd}->add( $self->{tail}, 0 );
return bless($self, $cn);
}
sub send
{
my $self = shift;
my $mess = shift;
return unless defined $mess;
#advance the head
my $id = $self->{memd}->incr($self->{head});
die("cache error") unless defined $id;
$self->{memd}->set($self->{name}."$id", $mess);
}
sub recv
{
my $self = shift;
return "empty" unless $self->length() > 0;
#advance the tail
my $id = $self->{memd}->incr($self->{tail});
die("cache error") unless defined $id;
return $self->{memd}->get($self->{name}."$id");
}
sub length
{
my $self = shift;
my $v = $self->{memd}->get_multi($self->{head},$self->{tail});
return -1 unless defined $v->{$self->{head}} && defined $v->{$self->{tail}}
&& $v->{$self->{head}} >= $v->{$self->{tail}};
return $v->{$self->{head}} - $v->{$self->{tail}};
}
1;
##############Example program####################
my $mq = new MemQueue("test2", "127.0.0.1:11211" );
my $c = 1000;
foreach my $i (0..$c){
$mq->send("message $i"x100);
}
foreach my $i (0..$c){
warn($mq->recv());
}
warn("len ".$mq->length());
jake memcached, perl, queue
Since I’ve been working on Thrudb I’ve become a big twitter user. So naturally I’ve created a Thrudb twitter account for people who are interested in tracking progress on the project. I also wanted to have the changeset tweeted whenever I commit code into subversion.
Subversion has a post-commit hook that will run a script when you commit something so I ended up finding twitvn which does what I wanted but only for trac based projects. Thrudb uses googlecode so it gets more complicated since we can’t install scripts on google hardware :) but they do support a nifty url callback that will post the commit info to you.
So off I went and whipped up a script just for googlecode projects that want to tweet their commits. Just put this in as a cgi on your webserver and tell googlecode the location (under Administration->Source tab). Here is an example of the output.
#!/usr/bin/perl
use strict;
use warnings;
use Net::Twitter;
use JSON::Any;
use Digest::HMAC_MD5 qw(hmac_md5_hex);
use LWP::Simple;
#Just update these
use constant SECRET_KEY => "SECRET_KEY_FROM_GOOGLE";
use constant TWITTER_USER => "TWITTER_USER";
use constant TWITTER_PASS => "TWITTER_PASS";
#check for defined digest from google
my $remote_digest = $ENV{HTTP_GOOGLE_CODE_PROJECT_HOSTING_HOOK_HMAC};
die("missing hmac digest") unless defined $remote_digest;
#fetch json message upto 100k
die("message too long") if $ENV{CONTENT_LENGTH} > 100000;
my $json_str;
read(STDIN, $json_str, $ENV{CONTENT_LENGTH});
#calc local digest and verify
my $digest = hmac_md5_hex($json_str, SECRET_KEY);
die("digests don't match") unless $digest eq $remote_digest;
#construct tweet and shorten changeset url
my $obj = JSON::Any->jsonToObj($json_str);
my $comment = $obj->{revisions}[0]->{message};
my $url = get "http://is.gd/api.php?longurl=".
"http://code.google.com/p/".$obj->{project_name}.
"/source/detail?r=".$obj->{revisions}[0]{revision};
die("problem shortening $url") unless defined $url && $url !~ "Error";
my $tweet = "svn: ".$comment." ".$url;
#shorten?
if(length($tweet) > 140){
$tweet = substr($comment,0,140 - 5 - (length($tweet) - 140))."... ".$url;
}
#tweet!
Net::Twitter->new(username=>TWITTER_USER(), password=>TWITTER_PASS() )
->update($tweet);
print "content-type:plain/textnn";
print "OKn";
jake googlecode, subversion, thrudb, twitter
When I was a kid my parents taught me to always have a plan and to always aim high. As a result, I have a master plan for life that I execute against. Things like losing weight, being a dedicated husband and father, coding nightly, retiring young, even this blog.
Like any kind of plan or project it must be flexible enough to change given new priorities or requirements. It also must have frequent results which can be measured (lbs lost, time spent, lines of code, posts and readership).
I’m probably not alone in this thinking, but I naturally apply agile methods to my life, just like I tend to apply them to software development. When some external problem comes up or I just get the urge to-do something new here’s my thought process:
- Does this change affect my other goals?
- How does this affect the priority of these goals?
- What goals can I postpone or drop to deal with this change?
The current economic situation has forced me to change some of my goals and priorities but having an agile approach to life, I believe, gives me the tools to handle it better.
Do you live an agile life too?
jake hacks, happiness, lessons, life
When I was a kid I thought my parents were always right. Whether it was the best way to dress or the best way to write a sentence for english class; if my parents said it was the better way then it was.
It wasn’t until 8th grade that I realized my parents really didn’t know much about a lot of things like clothes or music or grammar but they would have never admitted it. Eventually I learned to weigh my parents views as opinions that I respect, while at the same time using my own brain to decide on the right way for me.
If you are an entrepreneur, keep that in mind when you read something from people, companies or bloggers with authority. If you find yourself always accepting what they say and do as correct then you are probably like me in 7th grade.
jake business, lessons, life, web
I recently noticed the same names popping up again and again on the del.icio.us popular feed. So I pulled the last 5k posts and these are the most influential del.icio.us users:
I imagine these users have a knack of finding good content as well as having a large del.icio.us network.
Here’s how I did this:
Google feed api to fetch the last 5k users (you need to be authenticated to get this so i saved it from my browser)
http://www.google.com/reader/atom/feed/http://del.icio.us/rss?r=n&n=5000I then cat’d the results into this perl script: script.pl < popular.rss
#!/usr/bin/perl
my %users;
while(<>){
my ($user) = ($_ =~ /<name>([^<]+)/);
next unless defined $user;
$users{$user}++;
}
foreach my $user ( sort{ $users{$b} <=> $users{$a} } keys %users ){
print "$user\t$users{$user}\n";
}
jake del.icio.us, fun, stats
![[del.icio.us]](http://3.rdrail.net/blog/wp-content/plugins/bookmarkify/delicious.png)
![[Facebook]](http://3.rdrail.net/blog/wp-content/plugins/bookmarkify/facebook.png)
![[Reddit]](http://3.rdrail.net/blog/wp-content/plugins/bookmarkify/reddit.png)
![[Slashdot]](http://3.rdrail.net/blog/wp-content/plugins/bookmarkify/slashdot.png)
![[StumbleUpon]](http://3.rdrail.net/blog/wp-content/plugins/bookmarkify/stumbleupon.png)
![[Twitter]](http://3.rdrail.net/blog/wp-content/plugins/bookmarkify/twitter.png)