THIRD RAIL

Memcached based message queues

February 26th, 2009

Comments

Distributed caching and message queues are basic building blocks of distributed computing. Memcached is clearly the best of breed for caching. For message queues however the choices are many and vary in features, reliability and ease of use.

I needed a simple and fast queue with only basic reliability (lost messages don’t cost much). Memcached offers some special operations that make it possible to build queues… all you need is a couple counters. Here’s the basic idea:

Memcached queue design

First create head and tail keys to track the unread messages in the queue. Then using the atomic incr() operation memcached provides, increment the head and use that number as the key for a new message in the queue. Increment the tail and use that the key of the message of the next message to read. You can calculate the current queue size by subtracting the head and tail counters.

On the downside you may get no messages when you call recv() if the cache is full. Also, there is also no way of knowing what happens to a message once you recv() it. On the upside you can drop in a service like memcachedb for persistent storage.

There are similar queues built using memcached like sparrow.

Here is the perl implementation, should be easy to port to any language memcached provides clients for…

package MemQueue;

use strict;
use warnings;

use Cache::Memcached;
use constant PREFIX => "MEMQUEUE_";

sub new
{
    my $cn      = shift;
    my $name    = shift;
    die("queue name required") unless defined $name;

    my @servers   = @_;
    my $self      = {};
    $self->{name} = PREFIX().$name;
    $self->{head} = PREFIX().$name."_head";
    $self->{tail} = PREFIX().$name."_tail";
    $self->{memd} = new Cache::Memcached( servers => @servers );

    #create queue
    $self->{memd}->add( $self->{head}, 0 );
    $self->{memd}->add( $self->{tail}, 0 );

    return bless($self, $cn);
}

sub send
{
    my $self = shift;
    my $mess = shift;
    return unless defined $mess;

    #advance the head
    my $id = $self->{memd}->incr($self->{head});
    die("cache error") unless defined $id;

    $self->{memd}->set($self->{name}."$id", $mess);
}

sub recv
{
    my $self = shift;
    return "empty" unless $self->length() > 0;

    #advance the tail
    my $id = $self->{memd}->incr($self->{tail});

    die("cache error") unless defined $id;

    return $self->{memd}->get($self->{name}."$id");
}

sub length
{
    my $self = shift;

    my $v    = $self->{memd}->get_multi($self->{head},$self->{tail});

    return -1 unless defined $v->{$self->{head}} && defined $v->{$self->{tail}}
    && $v->{$self->{head}} >= $v->{$self->{tail}};

    return $v->{$self->{head}} - $v->{$self->{tail}};
}

1;

##############Example program####################
my $mq = new MemQueue("test2", "127.0.0.1:11211" );

my $c = 1000;
foreach my $i (0..$c){
    $mq->send("message $i"x100);
}

foreach my $i (0..$c){
    warn($mq->recv());
}

warn("len ".$mq->length());

Close Bookmark and Share This Page

Save to Browser Favorites / Bookmarks

Email This to a Friend

Copy HTML:

If you like this then please subscribe to the RSS Feed or Email Feed.

jake memcached, perl, queue

memcache++ – a c++ client for memcached

June 2nd, 2007

Comments

I’ve been using memcached for a couple years now and it’s by far one of the most useful open source tools. It’s written by Brad Fitzpatrick of livejournal, and used by everyone from, yahoo to facebook.

It so good because does one thing and it does it very well: It provides a distributed in memory cache.

Its super fast and super reliable, and it makes speeding up your site or application simple.

The server is written in c and uses libevent which is an highly scalable, asyncronous event based polling library which allows it to support tens of thousands of concurrent connections.

The client protocol is compact and there are clients in almost every language. The client libraies do most of the work since they can compress the data that goes into memcached and maintain multiple connections to different memcached instances running on your network (this is why the cache is distributed).

One of the few problems I’ve had with memcached have todo with the c based memcache clients libmemcache and apr_memcache.

libmemcache has problems corking the socket in linux, since it’s written for bsd. I’ve had reliability problems under high load where the client garbles the protocol.

apr_memcache works but the memcached protocol isn’t fully supported and it requires apr, which is a dependancy I’d rather not build into a non apache service.

Plus, I really wanted a OO api to use since I’m writing in c++ not c.

So I built memcache++ about a year ago, based origionally on a now defunct c++ client called memcachedpp.

Now that we have launched third rail, I now have a place to release memcache++ so grab it here:

http://3.rdrail.net/code/memcache++

Please send me a mail if you have problems with it, or if you are using it. I’m using it on a few projects with great success!

-Jake

Close Bookmark and Share This Page

Save to Browser Favorites / Bookmarks