I have been looking for simple-stupid ways of controlling floods of unwanted trafficfor ages; without memcached a cluster of servers gets worse at detecting the problem the bigger the cluster. indexing the created_at column is a big waste. Your solution is brilliant.

Howevah…

What about the bots you want? Sure, they’re not POSTing comment spam with links to V1@gr@ sites, or anything, but not every rogue bot is a POSTer. Whitelists/blacklists all, well they all suck, and heuristics User-agent bot detection suck only a bot, er, a bit less.

Trying to separate the sheep from the wolves is a challenge indeed.

Wouldn’t it be cool to make a little non-human detector based on patterns of behavior? Sure, the wget script would be simple: same IP, same User-agent, same rate. But detecting the smarter ones (like ones I have written in past, dark and evil days) that snarf up content, but irregularly (using rand()) and with innocent user-agents, but which happen with uncanny regularity.

Some patterns are intentional (Googlebot has a well-known list of IPs it comes from) and therefore helpful. Anything violating robots.txt rules is dead meat. Any IP that doesn’t also get the images on your page is either a) the last remaining Lynx user, or b) a bot. But it’s the tying this all together, especially in a large, clustered environment that makes the problem hard.

I think memcached can be used to aggregate this information in the same way you wrote about, and that should make the problem a) much simpler and immediate, and b) much lighter-weight.

And when it’s all done, all we need to do is figure out how to end spam for once and for all by turning the bots upon each other in some n-squared kind of way that makes the rest of us blokes just trying to focus on doing good things laugh with glee as the spambots self-destruct. Moo ha ha!

And if there’s not a good algorithm here, there’s gotta be a good B-movie plot.

Tom

Nice article!

Hey, not everybody needs images.

Check the code: validates_each(field, n => :create), it’s on creation only…