THIRD RAIL

Home > coding, thrift, thrudb > Announcing Thruqueue: Persistant message queue for Thrudb

Announcing Thruqueue: Persistant message queue for Thrudb

December 28th, 2007

I’ve just checked in a new Thrudb service that I’ve been working on for the past few days called Thruqueue. I’m sure you can guess by the name that it’s yet another message queue service. But this one has some great features that I think makes it stand out.

No hard limits – Create as many queues you like, send messages as large as you like, send as many messages as you like.

Persistant queues – Under the hood Thruqueue is exploiting Thrift’s powerful redo logging capabilities so queues are really managed logs, one log per queue. At specified intervals the logs are pruned to maintain disk space, this means the memory profile of thruqueue stays small since only a few items from each queue lives in memory at any given time.

Unique Queues – I’ve also added the ability to create unique queues which essentially means no duplicate messages can exist in the queue at once.

Fast! – I’ve done almost no performance optimization but my initial tests look very promising in 1 second I can write then read ~1200 small messages.

Thrift – Want a client in your favorite language? just run: thrift -favlanguage Thruqueue.thrift

Whats missing:

Replication – Do you really need this? I could hook this puppy up to spread but I’m not sure I see the benifit.

Redundancy – Throxy? TBD

I know that there are certainly a lot of message queues out there but all of then are either non-persistant, cost money, require an underlying rdbms, or cost money. Let me know what you think.

Close Bookmark and Share This Page

Save to Browser Favorites / Bookmarks

Email This to a Friend

Copy HTML:

If you like this then please subscribe to the RSS Feed or Email Feed.

jake coding, thrift, thrudb

Viewing 3 Comments

Jason 2 years ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

Hooking it up to Spread would be great for redundancy and replication. Multiple servers, not one a master. Push and pull to each with message synchronization across the whole. Though ordering would have to be sacrificed or at the very least made an issue for the readers.

reply edit reblog flag

jake 2 years ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

It takes the MD5 of the message and maintains a hash of keys in the queue. It rejects the message (silently) if a message already exists in the queue at that time. I could also make it absolutely unique by maintaining a bloom filter on the queue, so no messages will ever be duplicated to the queue (overkill).

Thruqueue is a stand alone service. runs on its own port, etc.

Thrudb was built to be made up of separate services so people can pick and choose the services they want.

reply edit reblog flag

http://3.rdrail.net

tobi 2 years ago 1 point
Please login to rate.

Do you already have an account? Log in and claim this comment.

Awesome. Unique Queues would be extremely useful in Shopify. How do they work? do they reject duplicates or remove existing entries and rather append the new one to the end of the queue?

Anyways, great addition but i'd almost wish this was a standalone product because it would also be supremely useful to legacy SQL apps.

reply edit reblog flag

http://blog.leetsoft.com