“The incompetency is staggering”
http://twitter.com/#!/crucially/status/201008727212560384
I couldn’t agree more. The post from Server Density does not go into the details of the Memcached deployment, but this description makes me think that they, indeed, are doing it wrong:
This eliminates the need for Memcached itself running on a separate cluster.
Separate cluster? It does indeed sound like Memcached-over-internet.
But why where they using Memcached in the first place? The answer is both funny and tragic:
the performance impact of the global lock in MongoDB 1.8 was such that we couldn’t insert our monitoring postback data directly into MongoDB – it had to be inserted into Memcached first then throttled into MongoDB via a few processor daemons
So, besides using MongoDB as a message queue they stick Memcached in front of it, using that too as a message queue, because MongoDB is too slow? I think this calls for:

