When I was a kid I thought my parents were always right.   Whether it was the best way to dress or the best way to write a sentence for english class; if my parents said it was the better way then it was.

It wasn’t until 8th grade that I realized my parents really didn’t know much about a lot of things like clothes or music or grammar but they would have never admitted it.  Eventually I learned to weigh my parents views as opinions that I respect, while at the same time using my own brain to decide on the right way for me.

If you are an entrepreneur, keep that in mind when you read something from people, companies or bloggers with authority.   If you find yourself always accepting what they say and do as correct then you are probably like me in 7th grade.

I imagine these users have a knack of finding good content as well as having a large del.icio.us network.

Here’s how I did this:

Google feed api to fetch the last 5k users (you need to be authenticated to get this so i saved it from my browser)
http://www.google.com/reader/atom/feed/http://del.icio.us/rss?r=n&n=5000I then cat’d the results into this perl script: script.pl < popular.rss

#!/usr/bin/perl

my %users;

while(<>){
        my ($user) = ($_ =~ /<name>([^<]+)/);
        next unless defined $user;

        $users{$user}++;
}

foreach my $user ( sort{ $users{$b} <=> $users{$a} } keys %users ){
      print "$usert$users{$user}n";
}

There is also mention of a fully asynchronous c++ client and server which I could really use…

The new home for thrift is here, check it out and use it with your project.

It’s pretty clear that Tues - Friday between 10am - 2pm PST are the “hot times” for popular blog posts.

Now, I didn’t filter out non-english posts and this doesn’t account for the time it took for the posts to get to the front page of these sites,  but I do think it’s clear posting late at night or on the weekends + monday is a bad idea.   Your post will most likley go unnoticed.

The service has been running now for about a week and it’s collected over 8 million tweets. I’ve run some stats on the lucene db and these are the top 100 words with more than 3 letters :)

I decided indexing twitter is a great application for thrudb, specifically the thrudex service. Thrudex is essentially a Thrift service for CLucene with some special sauce added. If you’d like to read about the inner workings read this.

Anyway, I whipped up a demo (in perl) for a realtime twitter search and have indexed a few days of tweets (over 3 million!) . check it out here.

One of our regular contributers, Thai Duong, was kind enough to port it to python+django for you new school folks.
*Note* this is running on a single dev box, so be forgiving… It’s currently polling the public timeline feed so it’s not going to catch every tweet. but It captures ~85%
We’ve added the code as a tutorial for thrudb here. Take it and build your own service… Any takers on building a ruby version or a cross site social search aggregation ala friendfeed?

What makes it stand out is really the DataStore API and Admin Interface. These two are really a game changer.

The DataStore API is a BigTable interface obviously. Without this we would just have another hosted scaling appserver solution without a database. The fact that the data layer is proven and ubiquitous just makes development/deployment simple. Frankly it scares me to think of how low the bar is to deploy an app now.

I remember when I was out of college I realized that most people thought programming was HARD! I secretly enjoyed the fact that, little did they know, programming by itself is really pretty simple, the hard part was getting something out the door. The fact that it’s getting so easy exposes our little secret.

I think I’ll get used to it, just need to look on the bright side. More focus on creative ideas.

That said, App Engine doesn’t fit everything. BigTable still doesn’t include any free text search which i think is ridiculous. I guess someone will write a python lib for creating an inverted index on BigTable. It also doesn’t fit backend services that crunch or enrich data. So I think mostly open social folks users will use it.

This does put a damper on a lot of startups that have been building user facing apps. I think it’s going to take some time for everyone to adjust but it’s all for the best i hope.

As for me, my startup is thankfully primarily a data enrichment service so App Engine isn’t a good fit. It also makes me think thrudb could pretty easily emulate the DataStore API so apps could be moved else ware.

Well done Google.

I recently read a great book called “Fooled By Randomess” by Nassim Taleb, that talks about the role randomness plays in life and specifically financial markets.  Nassim’s success as a trader came from his acceptance that failure will always occur and that you must place yourself in the position to expect and capitalize on those random failures.

I think this idea is applicable here since most successful entrepreneurs fail many many times before they succeed. In fact they often succeed because they bootstrap themselves from failures (I hope to be one of these).  The best example of this is James Dyson’s story where he failed hundreds of times before succeeding at creating his famous vacuum.

That’s not to say that you can’t be successful when you are young, look at Mark Zuckerburg or even The Million Dollar homepage guy, but these two aren’t exactly the norm.  I imagine if you suggested to them luck was why their ideas took off over everyone elses they would be terribly insulted.  We all want to believe deep down that we know exactly how to build the next big thing, I sure thought that.  Athletes are told to envision the goal, the free throw, the tackle.  This is valuable but by no means will help when it’s raining on the day of the game.

My 1 year old loves this show called The Wiggles. It was started by a couple of ex-rockers from Australia.  They each wear a different color shirt and sing silly songs about their dog, sleeping and buying apples.   They have earned ten multi-platinum awards for sales of over 17 million DVDs and four million CDs!   What a great story… Their pub rock band disbanded after little success then changing careers to become educators they hit upon a children’s rock band that’s now a worldwide phenomenon.

As I see it the key to success is perseverance.  Failing… getting back on the horse and trying again.  Only those that stop playing the game truly fail.

Here’s an example of how simple it is to write a basic http server.

#include <sys/types.h>
#include <sys/time.h>
#include <sys/queue.h>#include <stdlib.h>

#include <err.h>
#include <event.h>
#include <evhttp.h>

void generic_handler(struct evhttp_request *req, void *arg)
{
        struct evbuffer *buf;
        buf = evbuffer_new();

        if (buf == NULL)
            err(1, "failed to create response buffer");

        evbuffer_add_printf(buf, "Requested: %sn", evhttp_request_uri(req));
        evhttp_send_reply(req, HTTP_OK, "OK", buf);
}

int main(int argc, char **argv)
{
    struct evhttp *httpd;

    event_init();
    httpd = evhttp_start("0.0.0.0", 8080);

    /* Set a callback for requests to "/specific". */
    /* evhttp_set_cb(httpd, "/specific", another_handler, NULL); */

    /* Set a callback for all other requests. */
    evhttp_set_gencb(httpd, generic_handler, NULL);

    event_dispatch();

    /* Not reached in this code as it is now. */
    evhttp_free(httpd);

    return 0;
}

But is it fast? check out the benchmarks (on my laptop):

ab -c 1000 -n 10000 http://localhost/

Apache2: Requests per second: 1274.14 [#/sec] (mean)

libevent: Requests per second: 1584.37 [#/sec] (mean)

Kickass!